Big Data and data management in capital markets
Arun Varadarajan is head of client services, banking and financial services at iGate Technologies
Big data has gained significant momentum in driving innovative business models, creating new customer service paradigms and contextualising transaction data in real-time across multiple industries in recent years, writes Arun Varadarajan. Enterprises continuously produce huge amounts of transactional data for their customers, operations and suppliers. With rapid proliferation in adoption of smartphones, laptops and tablets, customers and employees add large volumes of data to the enterprise data stack.
Companies across various segments are increasingly looking at leveraging these new data sets that can help unlock the value of data beyond operational efficiency and post activity analysis. The availability of Big Data as well as Big Data technology allows for newer applications such as real time actions on changing consumer behaviour or real time customising of products and services based on events or faster analysis of large volumes of data to predict risk.
Big Data processing concepts now enable faster assimilation of data from social media sites, web sites, discussion boards, documents, and multi-media content. These data sets typically are characterised by volume, velocity, variability and variety – inherently posing significant data management challenges.
The capital markets industry continues to be amongst the top data driven industries. Electronic trading generates millions of market messages during a given day. Regulatory environment and risk management practices further mandate the banks and financial services firms to capture, store, and analyse data that spans over multiple years. With diminishing returns in high-frequency trading, focus has shifted from high-speed trading to looking for patterns in large volumes of market data for financial information and use cases. Financial services firms can leverage big data to deliver business insights and create new trading strategies.
Organisations in the financial services industry have recently embarked on creating Big Data Ecosystems that can process market data for pricing, corporate actions for faster portfolio processing. Most financial services firms have invested heavily in EDM (Enterprise Data Management) systems that currently cater to structured data (market data, reference data, econometric data) and are looking to extend the EDM capabilities to other types of data (news, sentiments, traffic information, buying patterns).
The Enterprise CIOs must take the lead on leveraging these technologies for managing Big Data such as Apache Hadoop ecosystem and industry standard Hadoop distributions such as Cloudera, Hortonworks, and PivotalHD, among others.
There are three direct benefits of doing this in a systematic manner:
- First, it avoids future challenges of bringing this data into the EDM fold as faced today with market and reference data (historically sourced in a federated manner.) This will ensure that these new data sources will follow the overall data governance processes of the firm from the get go.
- Second, this will help firms start leveraging, experimenting with new data sources creating new use cases and application areas that will increase their ability to significantly impact the business. As an example Big Data presents the opportunity for financial services firms to augment their risk management and reporting systems for faster pricing calculations, consolidated global positions and on-demand Value-at-Risk calculations.
- Third, the cost of these platforms and their availability make an interesting case for crunching and handling large data sets effectively brings down the cost of data processing for even traditional data sources and structured data.
Data management practices relating to data governance, stewardship, data quality for these new data sources is complex and must be given due attention. Behind-the-scenes activities used to create actionable insight have the daunting challenges of collecting the data and processing the data and delivering data that can be trusted for making decisions. Buy-side firms want to collect both internal and external data that can lead to actionable analysis. Sell-side firms carve out a competitive advantage by managing large volumes of data related to market-making, trading strategies, surveillance and risk management issues.
Other opportunities for Big Data include enabling post trade analytics helping asset managers to evaluate key metrics like transaction costs, order execution performance and portfolio returns measurement in real time. Insights from market indicators, economic indicators, and sentiment analysis for stocks and events may be used to enrich the information set used by Investment and portfolio managers for investment and asset balancing decisions. Big Data capabilities also enable the enterprise to develop comprehensive check-points for KYC initiatives, fraud detection, investigation and prevention. We expect that the overall relevance of Big Data continues to increase manifold and will play significant role in defining the overall performance of firms in capital markets over the years.
