The Economist catches the Big Data wave

At Causata we spend all our time with Big Data and so it's nice to see the Economist (current issue) produce a special report (written by Kenneth Cukier) about the explosion of data and the technologies needed to process it.

The business of information management—helping organisations to make sense of their proliferating data—is growing by leaps and bounds. In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specialising in data management and analytics. This industry is estimated to be worth more than $100 billion and growing at almost 10% a year, roughly twice as fast as the software business as a whole.

In addition to the special report there's an interview with the report's author:

Part of the Economist article was driven by a report from O'Reilly called Big Data: Technologies and Techniques for Large-Scale Data.

O'Reilly produced a set of interviews to accompany that report. Here's the first:

The rest can be found here.

So you think machine learning is boring?

If you say the words 'machine learning' to people they either look confused or bored. Since the promise of Artificial Intelligence evaporated in the 1970s, machine intelligence seems to be one of those things that's a perpetual 20 years away.

But computer industry insiders know that many forms of machine learning are at work all the time. The most common and visible are recommendation systems like the one on Amazon.com that comes up with suggestions for other books you might like. But even that doesn't express the true power of state of the art algorithms.

But a helicopter doing backflips and a walking robot do.

Here's AnyBots walking robot Dexter taking his first steps. Dexter is unusual in that unlike other walking robots he's essentially unstable (just like a human). He needs to constantly adjust his 'muscles' to stay upright. And he wasn't programmed to walk like that, he was programmed to learn to stay upright and walk.



And here's the Stanford University Autonomous Helicopter doing some stomach churning maneuvers. The helicopter is flying itself and being instructed to perform tricks for the camera. It learned to perform these tricks by learning to emulate the trajectory of a real helicopter flown by a human.



And if you've seen Lord of the Rings or Troy, or played Grand Theft Auto IV then you've seen a different sort of machine learning in action. Human motion in stunts in the films, and by characters in the game, are driven by algorithms that were created via computer evolution.



There's a lot happening in machine learning: the next ten years should be very exciting.

Careers: London-based Visual Designer

Causata is looking for a talented Visual Designer to join its User Experience team in its Mayfair offices. You'll work closely with our Lead Experience Architect and a world class team of software developers on our state-of-the-art web-based analytics software. As part of this growing dynamic start-up you'll be actively involved in shaping the user experience as well as the conceptual development of the product.

Working across a range of activities from interface design to data visualization and marketing collateral you'll have the opportunity to stamp your creative mark on a major piece of software.

You will need to be a strong conceptual thinker with an aptitude for problem solving but, above all, you'll be obsessive about your craft and the quality of your work.

Key Responsibilities:

• Participate in the conceptual development of the product range
• Create and maintain a consistent look and feel for the software and brand
• Design and production of screen layouts, user interface elements and pattern library
• Work with developers to deliver high quality product releases
• Produce design specifications and documentation
• Production of creative assets for ongoing marketing activities

You'll need:

• Impeccable design skills including layout, information design and typography
• An impressive online portfolio
• 2 years experience (website design, user interfaces, data visualisation or similar)
• Solid understanding of user-centered design principles and usability issues
• Excellent Photoshop, Illustrator and Flash skills plus a good working knowledge of InDesign
• Bachelor’s or Master's degree in Graphic Design, Digital Media Design or other visual specialty

Desirable:

• HTML and CSS for rapid prototyping
• ActionScript, JavaScript, Processing
• Interest in data visualization or statistics
• Experience creating Rich Internet Applications

So if you know your Tufte from your Few and want the chance to work on a revolutionary piece of software please contact us.

Careers: London-based core Java experts

Causata is looking to expand its Mayfair-based team of core Java experts working on our multithreaded and distributed data management platform. The current team consists of six software engineers working on a large code base that runs inside Amazon Web Services.

You need to have extensive experience in Java, and you need to be very comfortable with Java multi-threading. We are not looking for people whose experience of Java is snapping together some pre-built components, running them in an application server and writing some servlets: you'll need to be able to do those things, but most of our code is highly-optimized for our data management tasks (and volume).

If you have worked on very large scale applications which required high-availability and high-performance (such as trading systems, analyzing data from collisions in a particle accelerator, handling every click on Facebook.com) then contact us.

We are absolutely committed to automatic testing of all of our code, continuous integration and the use of agile development techniques. You need to think those things are a prerequisite for any serious software development effort.

We are a small team of skilled programmers and so we look for people who are good at working in a team and good communicators. If you like to work alone and prefer not to interact with a group of smart, fast-moving engineers, we are the wrong environment for you.

Since what we do involves statistics you need to be numerate. If you have a degree which included study of statistical methods it's a plus. If you went to a very good university it's a plus. Three of our team members have PhDs: having a good academic background is another plus.

We do use databases as part of our system, so knowledge of SQL and one of the open-source RDBMSs is a plus.

If this sounds interesting, please contact us.

Benchmarking - KDD and Netflix

John mentioned on an earlier post that machine learning and statistical methods are at the heart of what we do. They absolutely are. This is because we’re striving for the most accurate answers, with a latency that makes them commercially valuable. Creating the best engineered solution that delivers undisputed value is what we are about. In our world if it doesn’t provide a measurable performance improvement for the task at hand then it isn’t useful.

We apply the discipline of benchmarking in almost everything we do (...it may not surprise you that we operate as a true meritocracy at Causata... I absolutely believe that if somebody can do my job better than me then they should be doing it...!!) It keeps us focused and real. I recall some early days in 1997 and 1998 when I was privileged to work with two exceptional database marketing practitioners, Jacob Zahavi and Nissan Levin. They wrote some software at Urban Science that won the KDD cup for two years running. A significant difference between their approach and their more academic competitors was that these guys ruthlessly benchmarked everything they did against previous work, using large representative data sets. This prevented unfounded preferences in their methods arising through familiarity, discouraged misplaced pride, and encouraged the exploration of new techniques.

The frantic developments leading up to the Netflix prize on 26th July 2009 were a very public demonstration of the power of transparent benchmarking in developing engineering solutions for decision science. The ranking of performance clearly accelerated the validation of approaches that worked well and helped identify new hybrid opportunities.

It is huge fun working with talented and energized people in developing new solutions, but I have learned to respect benchmarking and champion challenge as processes that keep us grounded and drive engineering progress.

At Causata, we love machine learning and statistics

My small contribution to the widespread use of machine learning was the naive Bayesian POPFile email sorting program, and for Causata machine learning is at the heart of what we do.

Our CEO's former employers won the KDD Cup two years running. And the engineering team have worked on deep modeling of customer data and on recommendation systems.

If you read Programming Collective Intelligence and thought it was a light introductory text; if you've worked on a real, large-scale recommendations system; if you're more than familiar with supervised learning, multi-label classification, causality, Bayesian networks, SVMs, k-NN algorithms, or Weka; we'd like to hear from you.

Or if you're intimately familiar with R and know a z from a chi-square test, and agree that the sexy job in the next ten years will be statisticians then we also want to hear from you.

Members of the team created the first patent on exploration/exploitation and our advisors include Professor John Shawe-Taylor. Why not join us in Silicon Valley or London?

Careers: JavaScript Guru

Causata is looking for a very experienced JavaScript developer to work on our open source code base that has a potentially very significant industry impact. You'll be joining a team of two working on cutting edge JavaScript code designed to change the way the way online optimization software tools are deployed. You'll need to be very familiar with jQuery, YUI and other JavaScript technologies. You'll need to think that testing your code is vital. You'll write small, fast, clear, maintainable code.

The successful candidate's professional JavaScript skills, credibility in communications, energy, and ability to influence, will collectively all be critical to the success of this industry initiative.

Critically, you'll also be a good communicator. This role combines both hands-on development with evangelism. You should be comfortable blogging, answering developer questions on a mailing list, standing up at a conference and giving a presentation, and attending high-level customer meetings.

You'll have experience with one or more web analytics tools and be interested in keeping an eye on the market for online marketing and user experience software tools. Perhaps you'll also have worked for an online advertising service, or other company that needs to instrument web sites with JavaScript.

You'll be familiar with Flash and interested in learning about other technologies like AIR and Silverlight.

Ideally you'll have a bachelors degree in computer science (or something similar) and five years of real programming experience.