麒麟测试网 api_麒麟API我们如何构建数据管道

At Qraft Technologies, we’ve been refactoring our data-driven API service, Kirin API. Kirin API was initially developed by AI Research team providing different kinds of financial datas integrated with many other widely used database, covering corporate market data such as S&P Global’s Compustat database, Thomson Reuters’ Datastream and economic macro datas like FRED or Quandl APIs at one endpoint.

在Qraft Technologies，我们一直在重构数据驱动的API服务Kirin API。 麒麟API最初由AI Research团队开发，提供了与其他许多广泛使用的数据库集成的不同种类的财务数据，涵盖公司市场数据(例如S＆P Global的Compustat数据库，汤姆森路透的数据流)和经济宏数据(例如FRED或Quandl API)在一个端点。

As the demands of this project grows internally and externally, it has expanded into a commercial project from an internal pilot project, in charge of my (Research Platform) team into highly scalable and available service that can handle more traffics and more client.

随着该项目的需求在内部和外部的增长，它已经从内部的试验项目扩展到了商业项目，该项目由我的研究平台团队负责，成为可扩展的，可用的服务，可以处理更多的流量和更多的客户。

数据流 (Data Flow)

We started from the most simple architecture as the diagram below.Different kinds of financial datas are loaded in different databases and back-end layer layers provides different APIs to the clients. In our case, the client-side is the utility library that provides interface to python to call the API, and we have a roadmap to support other programming languages or platforms in the future.

我们从最简单的架构开始，如下图所示。不同种类的财务数据加载到不同的数据库中，后端层为客户端提供了不同的API。 在我们的例子中，客户端是实用程序库，它提供python调用API的接口，并且我们有一个路线图来支持将来的其他编程语言或平台。

There are some major problems with this architecture.Majority of end-point users are in-house python developers working on internally. However this requirement eventually would change to support not just Python but different interfaced clients such as R lanaguage or an Excel plugin to load data in spreadsheets. Another concern is the maintainence issue to support different versions of client interfaces in a single point.

这种架构存在一些主要问题。大多数端点用户是内部进行内部开发的python开发人员。 但是，此要求最终将改变为不仅支持Python，而且还支持不同接口的客户端(例如R lanaguage或Excel插件)以在电子表格中加载数据。 另一个问题是维护问题，无法在单个点上支持不同版本的客户端接口。

不可知服务层需求 (Needs for Agnostic Service layer)

Groups of clients have different requirements on how they interact with the data they want. Thus, the back-end layer should handle those complex requirements of each client and this cause serveral issues such as agility issues within the team, and high maintenance cost as the project proceeds.

客户组对如何与所需数据进行交互有不同的要求。 因此，后端层应处理每个客户的那些复杂需求，这会导致服务器方面的问题，例如团队内部的敏捷性问题以及随着项目的进行而产生的高昂维护成本。

There are some existing microservice design patterns and solutions for similar problems and those are Backend-For-Frontend(BFF) and GraphQL.

对于类似问题，存在一些现有的微服务设计模式和解决方案，例如Backend-For-Frontend(BFF)和GraphQL。

BFF和GraphQL (BFF and GraphQL)

BFF design pattern was one of the benchmark model for our API service. I am not going to cover the details about this, but the key aspect is that instead of providing a generic backend service, make a specific API that shares the business logic which is actually a part of client rather than backend.

BFF设计模式是我们API服务的基准模型之一。 我将不涉及这方面的细节 ，但关键的方面是，与其提供通用的后端服务，不如提供一个共享业务逻辑的特定API，该业务逻辑实际上是客户端而不是后端的一部分。

If BFF solved the problem in client-side’s aspect, GraphQL on the other hand, remains this design implementation on the server side. According to the official documentation, GraphQL is defined as:

如果BFF解决了客户端方面的问题，则GraphQL将这种设计实现保留在服务器端。 根据官方文档 ，GraphQL定义为：

GraphQL is a query language for your API and a server-side runtime for executing queries by using a type system you define for your data.

GraphQL是API的查询语言，并且是服务器端运行时，用于通过使用为数据定义的类型系统来执行查询。

GraphQL provides simplicity of the implementation of the backend server by providing a query language that client can request which specifies exact resources that client wants.

GraphQL通过提供客户端可以请求的查询语言来指定后端所需的确切资源，从而简化了后端服务器的实现。

BFF and GraphQL, both of them are looking for API layer how to provide an API that can be used by multiple types of clients requirements. The main difference is that BFF is geared to full-stack implementation, (Coupled BFF and client as one moduled component), GraphQL handles the complexity by proving an agnostic query and grants high degree of freedom to the client side. Our team has decided to use GraphQL

BFF和GraphQL都在寻找API层，如何提供一种可以被多种类型的客户需求使用的API。 主要区别在于BFF适用于全栈实现(BFF和客户端作为一个模块组件耦合)，GraphQL通过证明不可知查询来处理复杂性，并为客户端提供了高度的自由度。 我们的团队决定使用GraphQL

那么，为什么选择GraphQL？ (So Why GraphQL ?)

First of all, BFF has several issues to be applied:

首先，BFF有几个要应用的问题：

• Agility issues across BFF and client (these roles are split in our case)
  BFF和客户之间的敏捷性问题(在我们的案例中这些角色是分开的)

• Hard to predict capacity required by individual client

  难以预测单个客户所需的容量

  (especially for heavy requests such as daily/monthly tickers for large number of securities)

  (尤其是对于大量证券的每日/每月报价等繁重请求)

• Single point of failure
  单点故障
• Additional deployment complexity
  额外的部署复杂性

On the other hand, GraphQL helps dealing with evolving APIs since high degree of freedom is granted to the client groups as each groups can declaratively interact with data in AST(Abstract Syntax Tree) structure.

另一方面，由于每个客户组可以声明性地与AST(抽象语法树)结构中的数据进行交互，因此GraphQL有助于处理不断发展的API，因为客户组具有较高的自由度。

实作 (Implementation)

Currently, Kirin API service architecture using GraphQL looks as below.

当前，使用GraphQL的Kirin API服务体系结构如下所示。

Each client group requests data to API gateway(GraphQL server) and corresponding resolvers fetch data from original datasource. When data is fetched, postprocesses are handled by the asynchronous service workers and returns the result to the clients.Our CICD, Automated pipeline runs scheduled batches to check any updates on the datasources and validates integrity of the data and ensures each group of client versions are use up-to-date.

每个客户端组向API网关(GraphQL服务器)请求数据，相应的解析器从原始数据源获取数据。 提取数据后，异步服务工作程序将处理后处理并将结果返回给客户端。我们的CICD自动化管道运行计划的批处理，以检查数据源上的任何更新并验证数据的完整性，并确保每组客户端版本均为使用最新的。

缓存策略 (Cache Policies)

Resolvers in GraphQL runs independently so duplicated contexts between requests must be preliminarily checked. We inquiry if the data exists in the cache, and only fallbacks to original datasources when the data is not available in the cache.

GraphQL中的解析器独立运行，因此必须预先检查请求之间的重复上下文。 我们查询缓存中是否存在数据，并且仅当缓存中不可用数据时才回退到原始数据源。

— — — — — — — — — — — — — — — — — — — — — — — — — —

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —

We are stepping up our efforts to launch this service as a fine product, and wish this first product of our team could help many quant and AI researchers in finance, and furthermore successfully settle as a good fundamental of our research platform in the future.If you want to know more about Qraft Kirin API, Research Platform team is looking for enthusiastic team members to join us. We have alot of opportunities.Apply now.Trust me!

我们正在加紧努力，以优质产品的形式推出这项服务，并希望我们团队的第一款产品能够帮助许多金融领域的量化和人工智能研究人员，并进一步成功地成为我们研究平台的良好基础。您想进一步了解Qraft Kirin API，研究平台团队正在寻找热情的团队成员加入我们。 我们有很多机会。立即申请，相信我！

We believe highly-efficient data modification and research techniques can innovate the global finance business.

我们相信高效的数据修改和研究技术可以创新全球金融业务。

Join now!Qraft Technologies

立即加入！ Qraft Technologies

Team Research Platform

团队研究平台