At the time bitly launched it seemed a little frothy, we already had TinyURL. Was this what the world needed: a URL shortener so when Web surfers shared their endless musings through social-media they could shorten the links to their content?

After all, TinyURL allowed you to paste in a very long web address then provided you with a short code that redirected to it.

In the years since its birth, bitly has more than proved its business model. People used to ask what is the value of a service that takes a long Web address and makes it shorter? The answer was several million dollars.

The main activity bitly added to life online was and is URL tracking: in other words, we get a number for how many people clicked each link and it also keeps a copy of each page.

Hilary Mason’s job, as Chief Scientist at bitly for four years, was to play with this information, looking at data about what people click on, trying to figure out what it says about human behaviour and communication.

A third of her time was spent looking for interesting events, trends or visualizations. The other two thirds she focused on translating models and equations into functional systems.

She discovered the rather arcane fact that iPad, and other tablet owners are mostly likely to use their computers most often before bed, and while people like to share breaking news and current events, they’re more likely to read inane gossip, watch sneezing pandas or play online games.

Inane maybe but this data has its adherents who are willing to pay a lot of money for information.

She has also received a few awards including the TechFellows Engineering Leadership award, she was profiled for Forbes magazine’s 40 under 40 Ones to Watch list and has been featured in numerous glossy magazines including Glamour, Fast Company and Scientific American.

A former member of Mayor Michael Bloomberg’s Technology and Innovation Advisory Council not surprisingly she features prominently as one of the most important people in the New York Tech Industry.

After graduate school, Hilary joined Johnson & Wales University in Rhode Island as an assistant professor, but continued to program in addition to teaching and working on research and built a program that crawled job boards to determine which skills employers value, which helped Johnson & Wales explore ways to improve its curriculum.

Hilary is the founder of Fast Forward Labs, a machine intelligence research company, and the Data Scientist in Residence at Accel. She co-founded of HackNY is a member of NYCResistor, a non-profit company which helps talented engineering students find their way into the start-up community of creative technologists in New York City.

RM:

Hilary, I have read many things what data science is and what a data scientist does. It sounds a bit superhero-like to me because courses to train complete data scientists do not yet exist. Bits of disciplines exist in various departments around the country, and also in businesses, but as an integrated discipline it is only just starting to emerge.”

Are you able to untangle the web of information for me here and tell me what the central role of a data scientist is? Can anyone who is able to understand a little math, has plenty of logic and communication skills able to become a data scientist?

HM:

A data scientist does three things; first, they can build mathematical models of a system from messy data. Second, they can code. And third, they can ask good questions and communicate the answers back to non-technical people.

I think of ‘data science’ as a flag that was planted at the intersection of several different disciplines that have not always existed in the same place. Statistics, computer science, domain expertise, and what I usually call ‘hacking,’ though I don’t mean the ‘evil’ kind of hacking. I mean the ability to take all those statistics and computer science, mash them together and actually make something work.

And I think that last one is by far the most important. If you are the kind of person who can say, “I have some cool data. I really am curious about some questions about that data. I’m going to figure this out.” Then yes, you can do it.

RM:

Is data science one of those techniques which has out there for some time just waiting to be applied? Is it a branch of dynamic programming?

HM:

Data science is not a technique, it is a profession. It deserves a new and distinct title because it requires multiple skills in the same professional that have never existed together before.

Think of the job of the data scientist is asking the right questions, so if I ask a question like ‘how many clicks did this link get?’ which is something we look at all the time, that’s not a data science question. It is an analytics question. If I ask a question like, ‘based on the previous history of links on this publisher’s site, can I predict how many people from France will read this in the next three hours?’ that is more of a data science question.

Modern data science also bridges disciplines in a unique way, bringing together academia, the startup community, and, to a limited but growing degree, the corporate world.

RM:

What was it that made you think that ‘Wow, this is something I really need to explore and get into more’? Was it just intuition that this was the career path for you?

HM:

I am a computer scientist and have always had a keen interest in both algorithms and databases. It became clear to me in the last decade that the most interesting algorithms were those that worked on real data. I am curious, and I like understanding things.

The chance to work on a system that allows me to learn things about human behaviour that have never been understood before, with hard and interesting engineering challenges, was too good to pass up.

RM:

I understand that one of the products you developed was a real-time search engine which crunches data from its users is then able to spit back out again to searchers. As well as its computational relationship can it also pull answers from a range of approved references, databases and documents? Can it also visualize data and say, predict how many clicks a certain document will receive in the future?

HM:

We’ve did very successful predictive models of how many clicks a link will ever get, but didn’t found a use for that in any of bitly’s products.

One of the great things of the new crop of data-exploration software is that the inflection point of value in the data chain moves toward the right-out of heavy-duty processing systems that are expensive, complicated and must be maintained by IT, and into lightweight solutions almost anyone can use. Thus, the “priesthood,” ivory-tower academics or deep programmers, didn’t need to be consulted about every data-related question going forward.

The tools we created restricted the kind of analysis you’re able to do immediately, but they opened up a capability for analysis to a much wider population.The amount of learning you need to do to be effective does decrease from a multi-year process to a multi-hour process.

RM:

When modern computing started 50 years ago, people thought they would be able to ask a computer any question, and have it compute the answer. But it didn’t work out that way. Are we nearer that goal with what you are developing for business and will it get better as the platform is developed and more data sets are added?

HM:

I think we’re still fairly fair away from true AI. That said, there are people building tools that answer structured questions very efficiently, and there are even more opportunities in this space.

RM:

How much do you think you can sit down and figure out how something should work, assuming it is not something you have built before? Do you need to start writing code to understand what the problem is?

HM:

Personally, I like to think through a problem on paper or a whiteboard first, and then to write code to try it out, generally in ipython, then possibly to go back to paper and do some math, and then to re-write code, this time more robustly.

Python is my current programming language of choice, though I am not averse to C.

RM:

In general do you think data science is the next phenomenon? Regardless of this, what is the great promise of this data-exploration software for corporates and other organisations?

HM:

Data science offers a better picture of the truth of the world around us, which means that people can make more robust decisions. I don’t think it’s magic, just a cheap capability that everyone now has access to.

RM:

I’ve read that you see a lot of self-education with data science so where someone has two of the three essential components for conducting a productive inquiry and is able to teach themselves the third on their own. Is there any essential skill needed to do this? Is there a commonality you can highlight?

HM:

The essential commonalities are curiosity (motivation) and resourcefulness (ability to do things). If you have those two qualities, you can learn almost anything.

RM:

You mentioned that data science will enable people not having to consult the ‘priesthood’ or the ivory-tower academics and deep programmers about every data-related question. You obviously feel that academic computer science, the tech industry and industrial programming fail to meet in the right place?

HM:

In academia, you work on problems that will get you published. In industry, you work on problems that will make you money. Some questions are better explored in one of these contexts, and some

Startups are ahead of many large enterprises on this front, because many large corporations have heavy technology legacies and multiple layers of “priesthood” to negotiate.

The gaps between these communities’ take on data science can be characterized by their degree of openness. On the one hand, academia is known for its propensity to share information and conduct peer reviews. Startups are commercial in nature, but are just as likely to share findings with the open-source world as they are to consume open-source solutions.

Corporations, unsurprisingly, tend to be the most locked-down and opaque about their use of tools and their findings.

RM:

There have been a great number of tech start-ups in New York over the last few years do you think the NYC will soon outrun Silicon Valley for innovation?

HM:

The NYC versus Silicon Valley question is a false dichotomy. New York is a fantastic place to work in tech startups, and our culture is different than the valley. I think having more than one city with a strong tech culture will just lead to more creative projects in general.

There a lot of big opportunities around data management, data cleaning, helping people make better decisions from their personal data, sort of quantifying things about your life and understanding it in a very easy, frictionless way. I hope we see a lot more of that in the near future.