Entries Tagged as 'Big Data'

The 2014 Year in Review: Top 10 Blog Posts

Agile , Android , Big Data , DevOps , Malware , Secure Coding No Comments »

By Douglas C. Schmidt 
Principal Researcher

Douglas C. Schmidt In 2014, the SEI blog has experienced unprecedented growth, with visitors in record numbers learning more about our work in big datasecure coding for Androidmalware analysisHeartbleed, and V Models for Testing. In 2014 (through December 21), the SEI blog logged 129,000 visits, nearly double the entire 2013 yearly total of 66,757 visits. As we look back on the last 12 months, this blog posting highlights our 10 most popular blog posts (based on the number of visits). As we did with our mid-year review, we will include links to additional related resources that readers might find of interest. We also grouped posts by research area to make it easier for readers to learn about related areas of work. 

Read more...

Principles of Big Data Systems: You Can’t Manage What You Don’t Monitor

Architecture , Big Data No Comments »

By Ian Gorton 
Senior Member of the Technical Staff 
Software Solutions Division

Ian Gorton The term big data is a subject of much hype in both government and business today. Big data is variously the cause of all existing system problems and, simultaneously, the savior that will lead us to the innovative solutions and business insights of tomorrow. All this hype fuels predictions such as the one from IDC that the market for big data will reach $16.1 billion in 2014, growing six times faster than the overall information technology  market, despite the fact that the “benefits of big data are not always clear today,” according to IDC. From a software-engineering perspective, however, the challenges of big data are very clear, since they are driven by ever-increasing system scale and complexity. This blog post, a continuation of my last post on the four principles of building big data systems, describes how we must address one of these challenges, namely, you can’t manage what you don’t monitor. 

Read more...

Four Principles of Engineering Scalable, Big Data Software Systems

Architecture , Big Data 1 Comment »

By Ian Gorton
Senior Member of the Technical Staff
Software Solutions Division

Ian Gorton In earlier posts on big data, I have written about how long-held design approaches for software systems simply don’t work as we build larger, scalable big data systems. Examples of design factors that must be addressed for success at scale include the need to handle the ever-present failures that occur at scale, assure the necessary levels of availability and responsiveness, and devise optimizations that drive down costs. Of course, the required application functionality and engineering constraints, such as schedule and budgets, directly impact the manner in which these factors manifest themselves in any specific big data system. In this post, the latest in my ongoing series on big data, I step back from specifics and describe four general principles that hold for any scalable, big data system. These principles can help architects continually validate major design decisions across development iterations, and hence provide a guide through the complex collection of design trade-offs all big data systems require.

Read more...

Android, Heartbleed, Testing, and DevOps: An SEI Blog Mid-Year Review

Android , Architecture , Big Data , DevOps , Secure Coding , Testing 1 Comment »

By Douglas C. Schmidt 
Principal Researcher

Douglas C. Schmidt In the first half of this year, the SEI blog has experienced unprecedented growth, with visitors in record numbers learning more about our work in big datasecure coding for Androidmalware analysisHeartbleed, and V Models for Testing. In the first six months of 2014 (through June 20), the SEI blog has logged 60,240 visits, which is nearly comparable with the entire 2013 yearly total of 66,757 visits. As we reach the mid-year point, this blog posting takes a look back at our most popular areas of work (at least according to you, our readers) and highlights our most popular blog posts for the first half of 2014, as well as links to additional related resources that readers might find of interest. 

Read more...

The Importance of Software Architecture in Big Data Systems

Architecture , Big Data 7 Comments »

By Ian Gorton
Senior Member of the Technical Staff
Software Solutions Division

Ian Gorton Many types of software systems, including big data applications, lend them themselves to highly incremental and iterative development approaches. In essence, system requirements are addressed in small batches, enabling the delivery of functional releases of the system at the end of every increment, typically once a month. The advantages of this approach are many and varied. Perhaps foremost is the fact that it constantly forces the validation of requirements and designs before too much progress is made in inappropriate directions.  Ambiguity and change in requirements, as well as uncertainty in design approaches, can be rapidly explored through working software systems, not simply models and documents. Necessary modifications can be carried out efficiently and cost-effectively through refactoring before code becomes too ‘baked’ and complex to easily change. This posting, the second in a series addressing the software engineering challenges of big data, explores how the nature of building highly scalable, long-lived big data applications influences iterative and incremental design approaches.

Read more...