Showing posts with label Big Data. Show all posts
Showing posts with label Big Data. Show all posts

use big data to predict outbreaks of dengue fever and malaria

The research is aimed at understanding the spread of diseases in real-time in order to better deploy public health resources, said James Kaufman, public health manager at IBM Research in the IBM Almaden Research Center in San Jose, Calif.


But rather than just predicting the spread of a disease, the researchers at IBM, Johns Hopkins University, and the University of California at San Francisco are applying analytics from large data sets to see how changes in rainfall, temperature, and even soil acidity can dramatically affect the populations of wild animals and insects that carry the diseases. They’re also merging that information with other data, like airport and highway traffic, to further understand outbreaks.

To do this, IBM created an open-source modeling application dubbed the Spatio Temporal Epidemiological Modeler (STEM) that allows any kind of data to be quickly combined and correlated with disease data. The research could be important in understanding dengue fever, which is hitting places in Texas and Florida, thanks to the spread of mosquitos. The disease was once thought to be limited to the tropics or developing countries, but it’s showing up all over the world. Part of the reason is the rise of global transportation, trade, and climate change. Dengue fever has spread to more than 100 countries, and malaria is still responsible for a million deaths a year.

Analytics has been used to predict what you’ll buy next on Amazon or whether you’ll pay for an item in games like FarmVille. But it’s also useful in fields such as public health and disease research. Some parallels in the real world: Walmart uses sales data in its stores to predict when flu season starts and when to put more medicine on the shelves. And Google is able to predict the flu based on searches that people do about flu symptoms.

“We want to make it easier for people to gather and use data for research with open-source tools,” Kaufman said. “The thing that makes the greatest difference in the ability to make predictions about disease is the quality of your data. But most often there is a limitation in the data you have access to.”
In the past, it often took years to gather data from disease monitoring. But as the U.S. implements electronic health records across the country, it is becoming easier to access data in real-time, Kaufman said. And now that data can be crunched in the cloud, or web-connected data centers, and analyzed quickly. With STEM, scientists used population analytics, algorithms of disease paths, and powerful computing to build realistic and accessible models of these infectious diseases.

“Public health officials can’t afford to act on speculation during an epidemic. They need accurate and timely access to data to see what the potential spread of a disease might be for a given geographic region over a period of time,” said Kaufman.
With malaria, the researchers used the model and data from the World Health Organization. They were able to see how changes in local climate and temperature affected the spread of the disease. Now they can use that data to figure out where the next outbreaks will be. STEM is free and open to any scientist who chooses to build on its foundation in an open way. STEM 2.0 will be available on Oct. 15 through the Eclipse Foundation.

“There are a lot of tacit assumptions out there about how changes in climate will impact the distribution of diseases like malaria. This work suggests that things probably are not so simple. A change that has a huge effect on malaria transmission in one place might not be as important somewhere else,” said Justin Lessler of Johns Hopkins Bloomberg School of Public Health. “One of the nice things about open source projects like STEM is that now whoever wants to can download the model and start tweaking it, seeing if their own data or assumptions fundamentally change the results.”

“It is important to recognize the synergistic effort of theoretical and computational scientists, disease experts, and public health officials making a difference in how rapidly and effectively we fight infectious diseases,” said Simone Bianco of UC San Francisco’s Bioengineering and Therapeutic Sciences. “We have to be ready at the drop of a hat to parse through disparate data from global disease surveillance systems, conduct computationally intense research, and transfer our knowledge to public health officials to help them visualize population health, detect outbreaks, develop new models, and evaluate the effectiveness of policies.”

The research was published in the peer-reviewed journals Malaria and Theoretical Biology

use big data to predict outbreaks of dengue fever and malaria

The research is aimed at understanding the spread of diseases in real-time in order to better deploy public health resources, said James Kaufman, public health manager at IBM Research in the IBM Almaden Research Center in San Jose, Calif.

3 Ways Big Data, Supercomputing Change Weather Forecasting

"You can't control the weather." It's a phrase we often utter as we plan weddings, family picnics, and beach vacations. While it's true that we can't control the weather, scientists and technologists are working hard to improve our ability to predict it.


What that means for enterprise IT isn't just the ability to foresee disasters that could wipe out a data center. Granular and early forecasts can create business opportunities -- for example, alerting a brewery to supply distributors and points-of-sale greater inventory in anticipation of an unusually warm spring weekend in the Northeast. Or suppose a blizzard is due to hit Atlanta on Black Friday -- with enough notice, retailers could adjust their plans.

Beyond the burgeoning data services industry, weather has massive economic and safety implications. Weather Analytics, a company that provides climate data, estimates that weather impacts more than 33% of worldwide GDP, affecting the agriculture, tourism, fishing, recreation, and airline industries, to name just a few.

Uncertain weather conditions also impact small business owners, such as the local painter who can't complete his job on time when foul weather persists. In addition, public safety is of vital concern when officials aim to understand the impact of extreme weather events such as hurricanes, tsunamis, or wildfires. Costs associated with extreme weather across the world totaled more $125 billion in 2013, and the frequency of these events is on the rise.

Beyond the private sector, without specific data about a forthcoming event, governments may waste money casting a net that's too wide. For example, every additional mile of evacuation associated with an impending storm can result in millions of dollars in lost revenue, wages, and relocation expenses. And hospitals and emergency facilities that anticipate a severe storm that could shut down power could stock up on extra fuel for generators.

So while we're not able to control the weather, better forecasting will allow us to make more informed plans that can limit financial losses, provide new business opportunities, reduce government spending, and even save lives.
Unfortunately, improving our ability to predict the weather is challenging, both scientifically and computationally. Supercomputing has played a major role in enabling predictive models since the 1950s and remains at the cornerstone of today's weather and climate modeling. Constantly improving computational capabilities have allowed scientists and forecasters to produce results faster than ever while also investigating increasingly complex phenomena and producing specialized forecast products. From model performance to system and data management, weather prediction presents unique high-performance computing challenges.
[What percentage of your data center budget will be spent on maintenance vs. innovation in 2014? How about 2015? Tell us and enter to win a 32-GB Kindle Fire HDX.]

Supercomputing, along with big data, can meet the future demands of weather forecasting in three key areas:
1. Managing and utilizing enormous data sets: The volume and diversity of environmental data is increasing exponentially, placing great demand on the infrastructure to transport, manage, and store this data, and requiring ever-greater computational power for simulations that use it. This creates new opportunities for specialized services, developed with researchers in public and private institutions. One example is leveraging new sources of observation, such as sensors placed on automobiles. Imagine thousands of sensors in an urban area providing real-time meteorological information. Models are also evolving to analyze this tsunami of data and augment traditional physics-based simulations.

2. Increasing model resolution: Higher-resolution models are a critical element to better estimate the long-term state of climate systems and to improve weather forecasting, particularly for severe weather events. Recent simulations of Hurricane Sandy by researchers at the National Center for Atmospheric Research and the University of Illinois using Blue Waters supercomputers have zeroed in to a 500-meter resolution -- the equivalent of a few city blocks.

3. Addressing technology hurdles: As weather modeling and analytics become more data-intensive and computationally demanding, researchers must watch for performance bottlenecks such as memory, I/O, and interconnect latencies and bandwidths. Weather simulation requires thousands of microprocessors to run in parallel, pushing hardware and software to its scalability limits. In addition, scalable operating systems, compilers, and application libraries play an essential role in achieving sustained performance. Ultimately, the underlying technology infrastructure must be tightly integrated to support simulation and analytics workflows.
Infrastructures offering simulation and data-driven analytics capabilities to support routine execution of high-resolution forecasts will combine with advanced research to promote a whole new array of specialized meteorological services for public and private sectors. The future of weather forecasting requires capabilities we couldn't even conceive of when we began predicting the weather 64 years ago. Supercomputing innovation has so far kept pace with the demands of the community, and it is poised to offer new solutions in the years to come.

More..

3 Ways Big Data, Supercomputing Change Weather Forecasting

"You can't control the weather." It's a phrase we often utter as we plan weddings, family picnics, and beach vacations. While it's true that we can't control the weather, scientists and technologists are working hard to improve our ability to predict it.

Recover Data after an Unexpected Shutdown

If MongoDB does not shutdown cleanly [1] the on-disk representation of the data files will likely reflect an inconsistent state which could lead to data corruption. [2]
To prevent data inconsistency and corruption, always shut down the database cleanly and use the durability journaling. MongoDB writes data to the journal, by default, every 100 milliseconds, such that MongoDB can always recover to a consistent state even in the case of an unclean shutdown due to power loss or other system failure.

If you are not running as part of a replica set and do not have journaling enabled, use the following procedure to recover data that may be in an inconsistent state. If you are running as part of a replica set, you should always restore from a backup or restart the mongod instance with an empty dbPath and allow MongoDB to perform an initial sync to restore the data.
See also
The Administration documents, including Replica Set Syncing, and the documentation on the --repairrepairPath and storage.journal.enabled settings.
[1]To ensure a clean shut down, use the db.shutdownServer() from the mongo shell, your control script, the mongod --shutdown option on Linux systems, “Control-C” when running mongod in interactive mode, or kill $(pidof mongod) or kill -2 $(pidof mongod).
[2]You can also use the db.collection.validate()method to test the integrity of a single collection. However, this process is time consuming, and without journaling you can safely assume that the data is in an invalid state and you should either run the repair operation or resync from an intact member of the replica set.

Process

Indications

When you are aware of a mongod instance running without journaling that stops unexpectedly and you’re not running with replication, you should always run the repair operation before starting MongoDB again. If you’re using replication, then restore from a backup and allow replication to perform an initial sync to restore data.
If the mongod.lock file in the data directory specified by dbPath, /data/db by default, is not a zero-byte file, then mongod will refuse to start, and you will find a message that contains the following line in your MongoDB log our output:
Unclean shutdown detected.
This indicates that you need to run mongod with the --repair option. If you run repair when the mongodb.lock file exists in your dbPath, or the optional --repairpath, you will see a message that contains the following line:
old lock file: /data/db/mongod.lock. probably means unclean shutdown
If you see this message, as a last resort you may remove the lockfile and run the repair operation before starting the database normally, as in the following procedure:

Overview

Warning
Recovering a member of a replica set.
Do not use this procedure to recover a member of a replica set. Instead you should either restore from a backup or perform an initial sync using data from an intact member of the set, as described in Resync a Member of a Replica Set.
There are two processes to repair data files that result from an unexpected shutdown:
  1. Use the --repair option in conjunction with the --repairpathoption. mongod will read the existing data files, and write the existing data to new data files. This does not modify or alter the existing data files.
    You do not need to remove the mongod.lock file before using this procedure.
  2. Use the --repair option. mongod will read the existing data files, write the existing data to new files and replace the existing, possibly corrupt, files with new files.
    You must remove the mongod.lock file before using this procedure.
Note
--repair functionality is also available in the shell with the db.repairDatabase()helper for the repairDatabase command.

Procedures

Important
Always Run mongod as the same user to avoid changing the permissions of the MongoDB data files.
To repair your data files using the --repairpathoption to preserve the original data files unmodified.
  1. Start mongod using --repairto read the existing data files.
    mongod --dbpath /data/db --repair --repairpath /data/db0
    When this completes, the new repaired data files will be in the /data/db0 directory.
  2. Start mongod using the following invocation to point the dbPath at /data/db0:
    mongod --dbpath /data/db0
    Once you confirm that the data files are operational you may delete or archive the old data files in the /data/db directory. You may also wish to move the repaired files to the old database location or update the dbPath to indicate the new location.
To repair your data files without preserving the original files, do not use the --repairpath option, as in the following procedure:
  1. Remove the stale lock file:
    rm /data/db/mongod.lock
    Replace /data/db with your dbPath where your MongoDB instance’s data files reside.
    Warning
    After you remove the mongod.lock file you must run the --repair process before using your database.
  2. Start mongod using --repairto read the existing data files.
    mongod --dbpath /data/db --repair
    When this completes, the repaired data files will replace the original data files in the /data/db directory.
  3. Start mongod using the following invocation to point the dbPath at /data/db:
    mongod --dbpath /data/db

mongod.lock

In normal operation, you should never remove the mongod.lockfile and start mongod. Instead consider the one of the above methods to recover the database and remove the lock files. In dire situations you can remove the lockfile, and start the database using the possibly corrupt files, and attempt to recover data from the database; however, it’s impossible to predict the state of the database in these situations.
If you are not running with journaling, and your database shuts down unexpectedly for any reason, you should always proceed as if your database is in an inconsistent and likely corrupt state. If at all possible restore from backup or, if running as a replica set, restore by performing an initial sync using data from an intact member of the set, as described in Resync a Member of a Replica Set.

Recover Data after an Unexpected Shutdown

If MongoDB does not shutdown cleanly [1] the on-disk representation of the data files will likely reflect an inconsistent state which could lead to data corruption. [2]
To prevent data inconsistency and corruption, always shut down the database cleanly and use the durability journaling. MongoDB writes data to the journal, by default, every 100 milliseconds, such that MongoDB can always recover to a consistent state even in the case of an unclean shutdown due to power loss or other system failure.

Cloud Computing, Mobile and Big Data

As a Computer Science Student I'm going to do my Research project on Big Data,Cloud Computing and Mobile Computing  as my first step i read this article and i would like to share with you my experience so read and get knowledge about these technologies



 we look with optimism to 2015 as Gartner predicts that worldwide IT spending will surpass $3.7 trillion dollars next year.  According to Peter Sondergaard, senior VP at Gartner, we will see a new trend of spending outside of IT departments. “Every budget is an IT budget,” said Sondergaard during the annual Gartner Symposium/IT expo.

The major influencers are corporate-wide adoptions of cloud computing, big data projects and use of mobile technologies.  Sondergaard gives the examples of how sales departments are using cloud-based CRM packages and marketing departments are increasing social media use.  This trend will only accelerate as different departments adopt more and more cloud and mobile technologies.

With growing technology demands, Gartner analysts also predict that 1.9 million IT jobs will be created in the United States by 2015.  Sondergaard believes that only one-third of job positions will be filled due to lack of skilled applicants.  So in the upcoming years, CIOs will be tasked with both supporting cloud and mobile technologies while developing their workforce.

These predictions solidify the importance for CIOs to look at outside data centers for collocation, managed servers and cloud hosting.  If outside department demands accelerate as Gartner thinks, IT departments will be challenged with trying to support multiple departments as well as their own demanding IT projects – and doing all of this with a lower skilled workforce.

Outsourcing your data center allows your IT team to focus on your business support needs and not focusing on managing a data center.  You automatically can have a fully knowledgeable support team that is in charge of your managed dedicated servers.

Colocation providers have the staff that understands data security, high availability, and the strict HIPAA and PCI policies, procedures and compliance regulations.  With someone else managing your data center, your IT staff team can spend time developing applications and supporting corporate wide needs and initiatives.
As we come to a close on 2014, do you agree with Gartner predictions for 2015?  What are your 2015 trend predictions?

http://resource.onlinetech.com/2013-it-spending-trends-cloud-computing-mobile-and-big-data-projects/

Cloud Computing, Mobile and Big Data

As a Computer Science Student I'm going to do my Research project on Big Data,Cloud Computing and Mobile Computing  as my first step i read this article and i would like to share with you my experience so read and get knowledge about these technologies