In the age of information overload getting access to data might be less problematic than finding something meaningful in it. How to analyse thousands, millions and billions of lines of text, numbers and documents?

In this article Media4Change presents you some useful tools to manage large sets of information.

To make sense of data, it is useful to start from extracting names and unraveling relations between them. Open Calais is a free tool which helps to find people, places, names and concepts in large document sets (containing all type of information, like news articles, blog postings, legal documents, etc). This tool can identify relations that may uncover facts and events which can lead to the story.

Document Cloud is connected to Open Calais. You upload documents and they are run through OpenCalais and stored in Document Cloud. Then, this program allows you to make the documents available for the readers of your story to explore. To make its exploration easier, you can create notes with highlight key passages that contain meaningful information for your story and embed them in your reporting (for example, the specifics sentences that verify a story from a large document of an audio transcription). You can also make the whole document set public, so anybody can explore it.

Similarly to these two tools, an open source tool called Overview might be the first dig in a data set. Overview analyses your text automatically and categorize it by key words. It organizes texts in concepts and collect them together, so when a journalist have a document set and is not really sure what is in it, he can have a lists of words or topics of what is inside and start getting a broad idea of the content. This tool also offers the possibility to create visualizations of the analysis work.

Other similar tool is Detective.io. It works the same way as Overview, exploring connections in collected data, storing these documents safely and allowing the journalists to share them with other colleagues. The difference is that this tool is made to be used in different kind of electronic devices, so it offers more mobility.

In an online course about investigative journalism organised by Columbia University, Anya Schiffrin, Director of the Technology, Media and Communications specialization, claimed that we are currently living in a golden age of investigative journalism. Among the arguments she used to back her statement, she pointed out the enormous amounts of information available online and the development of social media.

One of the tools investigative journalists use to make sense of social media messages is Banjo. This tool organizes social media content by time and location, so it gives the context to frame stories. Anybody can make a request to Banjo’s search engine to have all the content that people are publishing on social media, concerning a specific time and location. Magically, journalists have access to first-hand information and multimedia content from eyewitnesses of events. There is also the possibility to communicate with the eyewitness directly, in order to verify information.

Apart from analysis of big data sets, there is still many offline sources of information that need to be analysed too. Casefile is an application which, as its creators say, is useful for those who are working ‘on the ground’: getting material from other people and building up information map of their investigations. Casefile allows journalists to create visual maps of interconnected links. Then, these graphs can be analyzed.

Another source of information are web pages. Data Miner helps to save time collecting information from them. This is a simple and useful tool that can extract data automatically from web pages and organise it in the Excel spreadsheet, which considerably reduces time journalists has to spend on collecting information.

Reg Chua, Executive Editor at Thomson Reuters, said that journalism has always adapted new methods, techniques and technologies to keep pace. Now, this pace of change, he said, is really accelerating, because the increase of vast amounts of data that is been created everyday.

In this post we informed you only about few tools that are useful in managing Big Data and planning the investigation. More suggestions for journalists on how to keep the accelerating pace and improve quality and accountability of their work is coming soon.