Feed Me the Data
Welcome to the new regime of real estate where we're becoming increasingly analytical and data is the new cryptocurrency. In some respects, we can identify with Seymour in "Little Shop of Horrors" whose Venus flytrap's mantra was "Feed me, Feed me." This page is designed to help satisfy your appetite for data, providing links to hundreds of thousands of data series that should satiate your appetites. While many of these sources may be new to you as they were to me when I started this journey down data lane, I hope you will embrace the spirit of Robert Frost, when you come to a fork in the road to big data: "... I took the one less traveled by, " and that made all the difference.
After a bit of scrolling around, I am confident you will discover there's "So, so much data." The challenge we face it to convert these data into actionable decision support; to monetize it. If you're not a fledgling data scientist and want to see what's possible, go to my new "Show Me the Mapping" page which features map-based information portals. Either way you do it, I wish you Bon Appetit!
____________________________________________________
How "Feed Me the Data" Works
This page is organized into major sections based on type of data and/or information provided by the various sources. To access the sites, click on the logos; for my thoughts, click on the respective Overviews. There's more here than meets the eye so I encourage you to explore the Overviews before diving into the data.
____________________________________________________
Economic Indicators
Several characteristics of the real estate market punctuate the importance of monitoring macroeconomic data. While all of these sources provide national-level data, many of them also produce data at the metro, county, zip, and census tract level.
____________________________________________________
One of the major services provided by the Federal Reserve Banks of St. Louis's Economic Research (FRED) is the data portal that provides acces to some 788,000 US and international time series datasets drawing on over 100 unique sources. Users are able to download, graph, and track historical and current data series. To explore FRED data, go to FRED Online. At this point, you can Search for data by name or, if you know, the mnemonic code.
-> FRED Overview
____________________________________________________
The FRED Excel Add-in can be launched from within Excel and can be used to create standardized queries that can be customized for geographic areas and datasets. FRED has provided a brief INSTALL FRED video to walk through the installation process. The installation of FRED uses the downloaded FRED file to activate the Add-in in Excel. Once the categories have been selected, the user simply checks on the Get FRED Data icon and the data will be downloaded. After a file has been successfully set up and data downloaded, it becomes a template that can be updated over time by simply loading the file and clicking on the "Get FRED Data" again.
-> FRED Add-in Overview
____________________________________________________
The Board of Governors of the Federal Reserve System, the central bank of the US, provides a wealth of data, reports, and other materials as part of its charge of providing a safe, flexible, and stable monetary and financial system.These materials can be access from the home page, using the drop-down menu to select items of interest. To improve ease of access, the FRS created a Data Download Program (DDP) where users can download data related to selected statistical releases. To help avoid errors and improve ease of downloading, they have prepared a "Things you should know" guide.
-> Federal Res Overview
____________________________________________________
The Beige Book contains a compilation of information from each of Federal Reserve Bank providign information on current economic conditions. The information is drawn from its from the Fed's branch directors, coupled with interviews with key business contacts, economists, market experts, and other sources. Once the information a compiled, a national overview is written to help explain what's happening as well as emerging trends. This national perspective is supplemented by a similar commentary from each of the 12 Federal Reserve Bank Districts providing insight into market conditions a prospects across the country.
-> Beige Book Overview
____________________________________________________
The BEA is an agency of the Department of Commerce that produces statistics to help government and business decision-makers, researchers, and the American public understand economic performance in the US. The BEA produces a number of products and services to discharge its mission including Principal Economic Indicators. Through interactive apps users can print, save and export data as well as generate charts that are both robust and visually appealing. A "Navigating the BEA Interactive Data" video can be reviewed.
-> BEA Overview
____________________________________________________
The US Census Bureau provides access to a variety of data sets and surveys that can be helpful in market analysis. The data can be retrived by searching, or by using some of the pre-set categories the Bureau has established. An example of this is the Census Economic Indicators that can be downloaded in PDF or Excel format. To provide greater access to Census data, the Census Bureau has written a series of How-to Materials for Using data.census.gov data have been produced.
-> Census Bureau Overview
____________________________________________________
The Census Bureau releases County Business Patterns (CBP) on an annual basis, providing a series of economic data by industry disaggregated to the county level. The scope of the CBP releases has expanded beyond counties, beginning with national data, and then disaggregated by State, County, Metropolitan Statistical Area (MSA), Combined Statistical Area (CSA), ZIP code, and Congressional District levels. The data include the number of establishments, employment, first quarter payroll, and annual payroll.
-> County BP Overview
____________________________________________________
The Bureau of Labor Statistics (BLS) publish statistics on employment and unemployment, as well as occupational employment and wages, labor demand and turnover, and the dynamic state of the labor market. The US Bureau of Labor Statistics (BLS) has created online interfaces that can help select data by type using its Geographic Guide. The labor-related datasets are organized into four categories: 1) Inflation, Prices, and Consumer Spending, 2) Employment and Unemployment, 3) Pay and Benefits, and 4) Productivity.
-> BLS Data Overview
____________________________________________________
The mission of the Department of Transportation (DOT) is "To ensure America has the safest, most efficient and modern transportation system in the world, which boosts our economic productivity and global competitiveness and enhances the quality of life in communities both rural and urban." DOT's priorities include: keeping the traveling public safe and secure, increasing mobility, and ensuring the US transportation system contributes to the nation's economic growth.
-> BTS Overview
____________________________________________________
Housing Markets
There are a wealth of data sources for the housing market including both the private market, and the public market.
____________________________________________________
Private Housing Market
____________________________________________________
NAR produces and analyzes a wide range of real estate data that can help guide businesses, members, and households. In addition to Research Reports and economic forecasts, data are released on key housing statistics, including State & Metro data.
____________________________________________________
Zillow is a "real estate and rental marketplace dedicated to empowering consumers with data, inspiration and knowledge around the place they call home, and connecting them with the best local professionals who can help." The companies services span the full lifecycle of owning and living in a home: buying, selling, renting, financing, remodeling and more. To support buyers, Zillow has created an interactive map interface that allows buyers to search by market using additional filters to narrow down the search. In a parallel manner to how it handles buyers, Zillow provides support for renters.
-> Zillow Overview
____________________________________________________
Public Housing Market
____________________________________________________
The Department of Housing and Urban Development (HUD) provides access to the original Datasets generated by the Office of Policy Development and Research (PD&R-) sponsored data collection. This access to data is to support independent research that can advance understanding of the nation's housing conditions and needs.
- The 50th Percential Rent consist of annual county/metro area reports on median Rents generated from the Fair Market Rent Process for use in Success Rate Payment Standards.
- Under the Community Development Block Grant (CDBG) Program provides annual grants on a formula basis to states, cities, and counties. The data are published annually metropolitan area CDBG Income Limits for Section 8 low-income limits (i.e., 80% of median income) and can be downloaded by users.
- The HUD-USER interface links to the HUD EXCHANGE Low- and Moderate-Income Survey Data (LMISD) Sets where users can access census Block Group level
- HUD-USER provides access to Fair Market Rents (FMRs) that are used to set standard payments for the Housing Choice Voucher program, set initial renewal rents for expiring project-based Section 8 contracts, as well as rates for housing assistance payment (HAP) contracts, the HOME Investment Partnerships, Emergency Solution Grants program, Continuum of Care recipients, and Public Housing units.
- HUD-USER also features an FRM Query interfacee that can be used to select state and county, or a HUD Metropolitan Fair Market Area.
- HUD publishes an annual list of Maturing Subsidized Mortgages that are slated to mature over the following 10 years.
- HUD-USER provides access to detailed data on its Single-family mortgage portfolio. The data are downloadable in Excel files with some pre-formatting for summary analysis and include mortgage purchases and refinancings.
- HUD publishes HOME Income Limits based on estimates of median family income, with adjustments based on family size. The data are produced annually and accessed by state filters to select metro or MSA levels of geography.
- HUD identifies Low-Vacancy Areas for purposes of funding the Tenant-Protection Vouchers (TPVs) set-aside for at-risk households in low-vacancy areas.
- The Multifamily: Active Section 202 Direct Loans provides detailed data on the Section 202 Direct Loan program.
- The Multifamily Assistance and Section 8 Contracts Database was introduced in February 2021 with the anticipation of becoming a monthly series.
-> HUD-USER Overview
____________________________________________________
The Low-Income Housing Tax Credit (LIHTC) program was created by the Tax Reform Act of 1986. The program gives State and local LIHTC-allocating agencies approximately $8 billion in annual budget authority to issue tax credits. HUD collects LIHTC data at both the property and tenant level and publishes the resultant LIHTC database. The database contains data on some 50,000 projects accounting for some 3.23 million units placed.
-> LIHTC Overview
____________________________________________________
Public Real Estate Market
The Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system, is the primary system for companies and others submitting documents under Securities, Trusts, and Investment Company Acts. Access to EDGAR’s public database is free which allows users access to all financial information and SEC filings of public company’s, mutual funds, exchange-traded funds, variable annuties, and individuals. To help users access this information, the Office of Investor Education and Advocacy has created a guide to Using EDGAR to Research Investments
-> EDGAR Overview
____________________________________________________
Nareit is a membership organization that serves as a global network of real estate companies operating as REITs or as eligible publicly traded real estate companies. The REIT universe also includes public non-listed REITs and private REITs as described in the REIT Sectors publication.
- Nareit produces a comprehensvie monthly REITWatch Report provides readers with a comprehensive overview of theindustry.
- Nareit visitors can download monthly reports including Performance by Property Sector/Subsector. This download includes the number of REITs by property sector and sub-sector, current total returns monthly and year-to-date, dividends, and market capitalization.
- To help investors and other parties stay current, Nareit publishes a series of timely articles under its Industry News tab.
-> Nareit Overview
____________________________________________________
Real Estate Indicators
The real estate market is one of the key components of the overall economy. A number of government and private data sources offer free data on the housing market including the following sources.
The Census Bureau publishes data on Construction Activity including residential, commercial, and public. The Census Bureau define construction activity to include building construction and engineering projects (e.g., highways and utility systems).
- The Census Bureau reports on Building Permits at various levels of geography ranging from US and Regions, to Metropolitan Statistical Areas (MSAs), and County or Place.
- CB data for New Residential Construction reports national and regional data on the number of new housing units authorized by building permits; authorized, but not started; started; under construction; and completed.
- To support analysis and graphing, CB data can be retrieved in TTime Series/Trend data. The data are retrieved through a menu-driven query that allows the user to select the type, date range, units or values, and geographic areas.
- The CB also publishes data reporting the Value of Construction Put in Place Survey (VIP) provides monthly estimates of the total dollar value of construction work done in the U.S.
-> Building Permit Overview
____________________________________________________
|
____________________________________________________
This page presents a data portal that can be used to acquire data-currency; the building block for advances in real estate analytics. The data are drawn from a disparate sources reflecting the interdisciplinary and global nature of real estate.
I will be adding more data sources so check back. If you have suggestions, please Contact Me.
____________________________________________________
Environmental Attributes
The Environmental Justice Screening and Mapping Tool is a great resource for assessing environmental quality of small geographic areas. Sites can be accessed by address or by interfacing with an interactive map. Once the report is produced on-screen, the user can select from three types of reports: environmental indicators, demographic indicators, and EJ indices. To help users decipher the EJS reports, the EPA created a Glossary of EJSCREEN terms. Users can also download a detailed EJSCREEN User Guide that provides detailed instructions on how to use the site.
-> EJSCREEN Overview
____________________________________________________
The Water Quality Portal (WQP) is a cooperative service sponsored by the United States Geological Survey (USGS), the Environmental Protection Agency (EPA) and the National Water Quality Monitoring Council (NWQMC) that integrates publicly available water quality data from the USGS National Water Information System (NWIS) the EPA STOrage and RETrieval (STORET) Data Warehouse, and the USDA ARS Sustaining The Earth’s Watersheds - Agricultural Research Database System (STEWARDS). The Water Quality Data Elements User Guide is available to help orient users.
____________________________________________________
The Agency for Toxic Substances and Disease Registry (ATSDR) is a public health agency working in collaboration with the National Center for Environmental Health (NCEH). of the U.S. Department of Health and Human Services. The ATSDR hosts the CDC's Social Vulnerability Index (CDC SVI) that users to compare vulnerability levels across geographic boundaries. SVI 2014 and 2016 are available for download in shapefile format or as .CSV files. The downloads are available for state and county levels of geography. Documentation for the SVI is at CDC SVI 2016 Documentation
-> Toxic Substance & Disease
____________________________________________________
The mission of the Center for Disease Control (CDC) is to protect America from health, safety and security threats, both foreign and domestic.
-> CDC Overview
____________________________________________________
The FBI compiles crime data through its Uniform Crime Reporting system on local criminal activity for some 18,000 agencies representing most jurisdictions in the country. The data are reported monthly and posted to the website as annual compilations. The individual files span back to 1995 making it possible to apply descriptive a as well as time-series, trend, and predictive analytics.
- The FBI makes summary crime data accessible disaggregated to the reporting jurisdiction of counties, cities, universities, and other geographical areas through its cloud-based, user-friendly Crime Data Explorer.
- At Table 8: Offenses Know to Law Enforcement a visiter can click on a state and then scroll down to the reporting agency of interest and observe the number of crimes by category.
- To help ensure the crime statistics are uniformly reported, the FBI publishes Data Quality Guidelines that explains how to classify criminal offenses using standardized definitions.
-> FBI Crime Overview
____________________________________________________
The Global Terrorism Database (GTD)™ is a comprehensive, unclassified database providing information on terrorist attacks across the globe. The database is distributed through the National Consortium for the Study of Terrorism and Responses to Terrorism (START). The data collection and data definitions are documented in the GGTD Codebook available for download on the START website.
-> GTD Overview
____________________________________________________
Geospatial Applications
The Census Business Builder (CBB) is a cloud-based suite of services that incorporate key demographic and economic data to help users make locational and business decisions. The CBB app is available in two versions:
- The Small Business Edition is primarily for businesses who need key data for a specific type of business to make locational decisions based on an understanding of market potential.
- The Regional Analyst Edition is focused on users interested in understanding the character of broader service areas including chambers of commerce and regional planning staff. Rather than focusing on a particular business, it incorporates data for all sectors of the economy in a customizable, a user-defined region.
- Given the robust and flexible nature of the two versions, users are encouraged to view some of the CBB resources that can provide more details. These include a series of CBB Webinars, as well as some written guides.
-> BUSINESS BUILDER Overview
____________________________________________________
The Office of Policy Development and Research (PD&R) supports the U.S. Department of Housing and Urban Development's (HUD's) initiatives to help create cohesive, economically healthy communities. To further its mission, PD&R features several applications for accessing data.
- To encourage utilization of PD&R data, it presents Biennial Reports to education users of its data and research about the organization, what it does, and how it functions.
- PD&R's
Research Projects are documented and made accessible in three categories: Ongoing Research, Recently Completed Research, and Other Completed Research.
- The American Housing Survey (AHS)The American Housing Survey (AHS) is sponsored by HUD and conducted by the U.S. Census Bureau. The survey provides up-to-date information about the quality and cost of housing in the United States and major MSAs including: the physical condition of homes and neighborhoods, the costs of financing and maintaining homes, and the characteristics of people who live in these homes.
- The Survey of Construction (SOC) provides current national and regional statistics on starts, completions, and characteristics of new, privately-owned single-family and multifamily housing units and on sales of new single-family houses.
- The Survey of Market Absorption of Apartments (SOMA) uses the Census Bureau's Survey of Construction (SOC) as its sampling base. Each month, a sample of residential buildings containing five or more units is selected for SOMA. The initial 3-month interview collects information on amenities, rent or sales price levels, number of units, type of building, and the number of units taken off the market (absorbed).
- The Manufactured Homes Survey (MHS) provides data on the manufactured homes industry including key statistics and shipments. The datasets are available for download under the Census's Public Use Files.
- The Rental Housing Finance Survey (RHFS) provides a current and continuous measure of financial, mortgage, and property characteristics of rental housing properties in the United States.
- The American Healthy Homes Survey II (AHHS) was HUD, in cooperation with the EPA, surveyed homes in the U.S. to evaluate the presence of lead-based paint and lead-based paint hazards as well as water samples for lead, air samples for formaldehyde, floor dust samples for mold, wipe samples for pesticide residues, and presence of lead water pipes feeding the home (service lines).
-> OPD&R Overview
____________________________________________________
Global Data Sources
There is a wealth of global data sources available to support real estate decisions. This is a collection of sources providing data, overviews, and insights. Please note that many of the macroeconomic resources on this page also provide global data.
The World Bank Data site is designed to make it easy to find, download, and analyze global data. Users can search for individual indicator names, countries, and topics or combine them to customize a query. The search box is responsive and lists matching results as you enter names, countries, and topics. Results are presented in modifiable graphs. To access data by country, click on Country or Economy
-> WB Overview
____________________________________________________
The World Bank engages 189 member countries, staff from more than 170 countries, and offices in over 130 locations. As a unique partnership, it consists of five institutions working for sustainable solutions that reduce poverty and build shared prosperity in developing countries. In addition to data, the World Bank has many reports and other resources to help you stay abreast of global conditions and issues you can access by clicking on the link.
->WB Resources
____________________________________________________
OECD.Stat includes data and metadata for OECD countries and selected non-member economies. Users can access the data by keyword search, selecting data in a menu using popular queries or thems, or by access saved queries. The OECD site can be customised by registering for free to create a My OECD.
-> OECD.Stat Overview
____________________________________________________
OECD-iLibrary provides an extensive series of publications and research materials covering an array of topics related to the core mission of OECD. The iLibrary has created a publication portal (SDG) that serves as a gateway to the publications using a visualization to help users navigate through the myriad publications. The SDG network model illustrates the linkages among topics that can be explored through a hierarchical system
-> OECD iLibrary Overview
____________________________________________________
The Future Data Lab (FDR) functions as a consortium of acadamics and business partners supporting several missions including: spatial data services, development of tools for spatial data analysis, workflow based spatial data case studies, and training programs for spatial data science.
-> FDL Overview
____________________________________________________
The UN Human Development Programme provides access to data sourced from international data agencies that collect national data on specific indicators. The data are presented in two sections: a ‘Data’ page with access to over 150 global indicators and composite indices for over 190 countries; and, a ‘Country profiles’ page presenting HDI values, ranks, and data visualizations/trends for the HDI and other indicators.
-> UN-HDR UPDATED
____________________________________________________
The goal of Our World in Data is to make accumulated knowledge on the big problems facing the world accessible and understandable in hopes of helping identify solutions. These problems include poverty, disease, hunger, climate change, war, existential risks, and inequality. THe organization recognizes we need data to quantify the important issues and make the results accessible in an understandable using interactive data visualizations to help track how the world has changed and then, by summarizing the scientific literature, help us understand why it has changed.
-> Our World Data Overview
____________________________________________________
The European Statistical System (ESS) is the partnership between the Community statistical authority, which is the Commission (Eurostat), and the national statistical institutes (NSIs) and other national authorities responsible in each Member State for the development, production and dissemination of European statistics. The Eurostat Dashboard provides access to data and information organized into key themes.
-> Eurostat Overview
____________________________________________________
Macrotrends describes itself as the "Premier Research Platform for Long Term Investors." The Macrotrends website provides access to over 200,000 charts and underlying data that can be accessed via Search Bar or selecting on categories. The initial selection will provide snapshot images of the graphs related to the topic; clicking on the graph of interest opens a larger graph that can be modified in terms of time span, time bracket, and recessions.
-> Macrotrends Overview
____________________________________________________
Open Data: In Process
In addition to traditional data sources cited on this page, there is a dramatic increase in the number of "Open Data" portals hosted by cities, counties, and other jurisdictions across the country. For many agencies, these efforts have been on-going and provide rich longitudinal datasets. For others, the iniatives are more recent, having been mandated by the 2017-18 OPEN Government Data Act. In terms of scale, a search of the DATA CATALOG reveals almost 200,000 datasets that are accessible to the public.
____________________________________________________
____________________________________________________
|