Entries Tagged as 'Big Data'

Principles of Big Data Systems: You Can’t Manage What You Don’t Monitor

Big Data No Comments »

By Ian Gorton 
Senior Member of the Technical Staff 
Software Solutions Division

Ian Gorton The term big data is a subject of much hype in both government and business today. Big data is variously the cause of all existing system problems and, simultaneously, the savior that will lead us to the innovative solutions and business insights of tomorrow. All this hype fuels predictions such as the one from IDC that the market for big data will reach $16.1 billion in 2014, growing six times faster than the overall information technology  market, despite the fact that the “benefits of big data are not always clear today,” according to IDC. From a software-engineering perspective, however, the challenges of big data are very clear, since they are driven by ever-increasing system scale and complexity. This blog post, a continuation of my last post on the four principles of building big data systems, describes how we must address one of these challenges, namely, you can’t manage what you don’t monitor. 

Read more...

Four Principles of Engineering Scalable, Big Data Software Systems

Big Data No Comments »

By Ian Gorton
Senior Member of the Technical Staff
Software Solutions Division

Ian Gorton In earlier posts on big data, I have written about how long-held design approaches for software systems simply don’t work as we build larger, scalable big data systems. Examples of design factors that must be addressed for success at scale include the need to handle the ever-present failures that occur at scale, assure the necessary levels of availability and responsiveness, and devise optimizations that drive down costs. Of course, the required application functionality and engineering constraints, such as schedule and budgets, directly impact the manner in which these factors manifest themselves in any specific big data system. In this post, the latest in my ongoing series on big data, I step back from specifics and describe four general principles that hold for any scalable, big data system. These principles can help architects continually validate major design decisions across development iterations, and hence provide a guide through the complex collection of design trade-offs all big data systems require.

Read more...

Android, Heartbleed, Testing, and DevOps: An SEI Blog Mid-Year Review

Android , Big Data , DevOps , Secure Coding , Testing 1 Comment »

By Douglas C. Schmidt 
Principal Researcher

Douglas C. Schmidt In the first half of this year, the SEI blog has experienced unprecedented growth, with visitors in record numbers learning more about our work in big datasecure coding for Androidmalware analysisHeartbleed, and V Models for Testing. In the first six months of 2014 (through June 20), the SEI blog has logged 60,240 visits, which is nearly comparable with the entire 2013 yearly total of 66,757 visits. As we reach the mid-year point, this blog posting takes a look back at our most popular areas of work (at least according to you, our readers) and highlights our most popular blog posts for the first half of 2014, as well as links to additional related resources that readers might find of interest. 

Read more...

The Importance of Software Architecture in Big Data Systems

Big Data 7 Comments »

By Ian Gorton
Senior Member of the Technical Staff
Software Solutions Division

Ian Gorton Many types of software systems, including big data applications, lend them themselves to highly incremental and iterative development approaches. In essence, system requirements are addressed in small batches, enabling the delivery of functional releases of the system at the end of every increment, typically once a month. The advantages of this approach are many and varied. Perhaps foremost is the fact that it constantly forces the validation of requirements and designs before too much progress is made in inappropriate directions.  Ambiguity and change in requirements, as well as uncertainty in design approaches, can be rapidly explored through working software systems, not simply models and documents. Necessary modifications can be carried out efficiently and cost-effectively through refactoring before code becomes too ‘baked’ and complex to easily change. This posting, the second in a series addressing the software engineering challenges of big data, explores how the nature of building highly scalable, long-lived big data applications influences iterative and incremental design approaches.

Read more...

Addressing the Software Engineering Challenges of Big Data

Big Data No Comments »

By Ian Gorton
Senior Member of the Technical Staff
Software Solutions Division
(This blog post was co-authored by John Klein)

Ian GortonNew data sources, ranging from diverse business transactions to social media, high-resolution sensors, and the Internet of Things, are creating a digital tidal wave of big data that must be captured, processed, integrated, analyzed, and archived. Big data systems storing and analyzing petabytes of data are becoming increasingly common in many application areas. These systems represent major, long-term investments requiring considerable financial commitments and massive scale software and system deployments. With analysts estimating data storage growth at 30 to 60 percent per year, organizations must develop a long-term strategy to address the challenge of managing projects that analyze exponentially growing data sets with predictable, linear costs. This blog post describes a lightweight risk reduction approach called Lightweight Evaluation and Architecture Prototyping (for Big Data) we developed with fellow researchers at the SEI. The approach is based on principles drawn from proven architecture and technology analysis and evaluation techniques to help the Department of Defense (DoD) and other enterprises develop and evolve systems to manage big data.

Read more...