Data Management People, Tools and Processes Defined
The following narrative will focus on defining the people, tools and processes that comprise a critical part of the data management program and support the organization’s data management capability. It will utilize a common framework template show in Figure 1, below (from our previous blog in this series, Data Management Framework).
Figure 1: Data Management Framework
The purpose of utilizing the same diagram to depict the people, tools and processes is to allow the reader to overlay each consecutive diagram and identify how the people, tools and processes are aligned with one another and how they relate to others. The people, tools and processes defined in each diagram are meant to be exclusive lists of each respectively. The intent is to blog each role (person), tool and process once where its value to the organization is at its greatest. As your Company becomes familiar with the diagrams and the people, processes and tools, its staff can utilize the diagrams to draw relationships and where people, tools and processes will need to overlap to show the configuration that is sustainable and the right fit for your Company to meet its business strategies.
Data Management Roles (People)
Figure 3 below depicts the roles necessary to implement a comprehensive data management program, which should include roles from all ten (10) data management functions. The intent of this figure is not to depict each individual role that would perform tasks within each function, rather to show the exclusive roles defined by this framework and where they provide the most immediate value when developing the data management capability. Depending on the maturity of the data management program, multiple roles depicted below can be filled by one individual or may require multiple individuals for a given role. For example, in organizations that have a mature data management program, the role of data developer will be filled by multiple individuals, and can be further broken down into Data Developers for custom application development and Data Developers for Data Warehouse development. In organizations where there is minimal data warehousing and business intelligence management capabilities, one data developer may develop code to support the organization’s custom data application and data warehouse initiatives.
Additionally, some roles like Data Analyst which is placed under the Master Data and Data Warehouse Management functions assumes that the data management capability for the organization is very mature among all functions. In this scenario the responsibilities of the Data Analyst associated with data sourcing and curation are occurring within these functions at the source (Master Data Management) and along the data transformation processes (Data Warehousing) where data analysis is most important. However, if the organization has a less mature data management program, the data analyst role will likely exist within the Data Development Management function.
Figure 3: People involved in Data Management Functions
At the top of figure 3 above and within the Data Management Framework are the Chief Information Officer, Data Governance Office Staff and Project Management Office Staff. These resources are integral to a comprehensive data management program. The CIO is ultimately responsible for the data strategy for the organization and therefore is accountable for the development of quality data management capabilities. The Data Governance Office and its Staff will leverage the information contained in the Business and IT policies, procedures and standards, along with other data management function artifacts to implement its strategies and achieve its goals. It will also influence decisions related to the tools the Data Management program will utilize and processes of which the Data Management program will need to focus. Organizations with a mature data management program and a formalized Data Governance Office will budget for staff who will oversee and run the Data Governance activities. Data Governance Staff are responsible for managing data governance activities including developing a data governance charter, developing the data governance committees and workgroups, scheduling and documenting meetings, monitoring Data Management activities, and maintaining data governance policies, procedures, and processes. The Project Management Office (PMO) will provide project management capabilities for data management projects and make reports available to the data governance office when required.
Using Data Security Management as an example, the roles (people) for this function are described in the following chart. The roles include those common throughout the framework, those specific to Data Security Management and those included in the related concentric circle drawn around it and Data Operations Management. The term “Capacity Driven” in the table below means that the role may vary in the number of people performing it depending on resource requirements at specific point in time.
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| Chief Information Security Officer | Reports to the CIO and is accountable for the security capabilities of the enterprise.
|
Required | IT |
| Network Administrator | In charge of the design and implementation of network components that build and harden Network Infrastructure. | Required | IT |
| Network Security Staff | Implement the security processes and procedures defined within the data security management function to ensure data is protected at the network level. Access Level at the Network Level. | Required, Capacity Driven | IT |
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| Database Administrator | Develops standards to develop data artifacts; Access Control at the database level. Approves and migrates database changes to the various release environments. | Required | IT |
| System Administrator | Responsible for the installation of and updates to Business and IT Application Software; Access Control for Business Systems | Optional (May be Performed by Server Administrator Role) | IT |
| Compliance Analyst | Responsible for the interpretation of security regulations and analysis of technical security compliance. | Required | Business |
| Data Owner | Accountable for a specific set of data required to complete job functions. Dictates data classification and Business Data Policy Decisions. | Required | Business |
| Data Steward | Verifies Data is Stored and Transmitted based on data classification requirements | Optional/Capacity Driven | Business |
| Data Governance Staff | Apply Data Governance Policies, Procedures and Processes to enable a robust, cross-functional decision making process; Define success criteria and Monitor Performance against Defined Metrics | Required, Capacity Driven | Business and IT |
Depending on the data management needs, there will be collaboration among staff assigned to other data management functions. This may be in direct response to a request by a data governance committee looking to implement a new initiative that requires the input from individuals representing different data management functions. In other situations, it may be that Data Operations management needs to implement a new data encryption procedure which requires the input from Data Security Management. The purpose of the proposed effort, if well defined, will dictate the roles, processes and tools required to be successful. It must be stressed that any role defined within the Data Management framework must include a detailed description of its job duties, responsibilities, processes, procedures and standards for the work it is to perform to be effective; especially for any role that is capacity driven and is a candidate to be outsourced. If the person representing the role must rely on other people within the organization to define what and/or how to perform its duties while a project is underway, it is likely the effort will experience unanticipated delays, result in a patchwork of processes and procedures that are poorly defined and documented, and an implementation that will make data quality analysis, troubleshooting and enhancements more challenging and time consuming.
The other Data Management roles depicted in Figure 3 are presented in the following table. The roles assume the people who carry them out are specialized in fulfilling the role responsibilities. It is expected that the people performing these roles can provide input across multiple data management functions within their expertise and experience.
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| Server Administrator | Manages server operations to ensure availability and resilience of server infrastructure
|
Required, Capacity Driven | IT |
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| Operations Staff (Help Desk) | Supports end users to troubleshoots user function related issues
|
Optional, Capacity Driven | IT |
| Operations Staff (Service Desk) | Supports IT users to troubleshoot and mitigate infrastructure issues.
|
Required, Capacity Driven | IT |
| Data Developer | Responsible for developing data definition, manipulation code based on IT Standards | Required, Capacity Driven | IT |
| User Interface Developer (Application) | Develops UI Data Validation Code to Support the Capture of Business Data | Optional (COTS, SaaS), Required (Custom), Capacity Driven | IT |
| User Interface Developer (Business Intelligence) | Develops Data Presentation Artifacts (Dashboards, Public Web Reports)
|
Optional, Capacity Driven | IT or Business (Self-Service) |
| Technical Writer | Develops and Maintains policies, procedures and diagrams which define cross-functional business processes and procedures related to data management activities for the organization.
|
Optional, Capacity Driven | Business and IT |
| Librarian | Categorizes Data and Defines Document (Record) and Content Taxonomies; Maintains Data Catalog/Library
|
Optional | IT and Business |
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| Subject Matter Expert | Provides Business Knowledge input for various data management activities; Defines Business Process and Procedures | Required | Business |
| Business Analyst | Works within business units to gather business requirements for new change management needs, document business process flows, policies, procedures, change requests and monitors change management requests to completion. | Required, Capacity Driven | Business |
| Data Analyst | Acts as the intermediary between business analyst and data developer; Define Data Definitions and Investigate new Data Sources and Processes | Required, Capacity Driven | IT |
| Data Quality Analyst (Business) | Analyzes Presentation/Reporting Data for Quality (UAT); Provides Input to IT Data Quality Analyst | Required | Business |
| Data Quality Analyst (IT) | Analyzes Data Sources and Data Preparation Methods; Performs Data Quality Control Prior to Implementation; Can be performed by Data Analyst or Data Developer (Peer Review) | Optional | IT |
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| Report/Visualization Developer | Develops Reports, File Extracts and most Advanced Visualizations in Support of Operations and Analytics Teams
|
Required, Capacity Driven | IT or Business (Self-Service) |
| ETL Developer | Develops processes and procedures to extract, transform, load (ETL) and integrate raw source data from various enterprise applications for reporting purposes
|
Optional, Capacity Driven | IT |
| Data Scientist | Develops advanced statistical models to mine data and discover data associations for AI, descriptive and predictive analyses
|
Optional | IT and Business |
| Data Architect | Designs and Develops Data Architecture (OLTP, ODS, Staging, Data Warehouse) for Transactional and Reporting Data Solutions, Develops Data Architecture Models, Develops Source/Target Mapping, Designs System Data Integration Roadmap.
|
Required | IT |
| Role Name | Role Description | Required/Optional/Capacity Driven | IT versus Business Role |
| System Architect | Designs system architecture strategies to implement data architecture models. This role has extensive experience implementing data patterns utilizing key business system features.
|
Optional | IT and/or Business |
| Data Stakeholder | Business Analyst and Business Unit Managers who will be impacted by changes to Data
|
Required (Depending on Influence can Sink Data Initiatives if concerns are perceived to be dismissed. | Business |
| Audit Analyst | Analyzes and Confirms Adherence to Business and IT policies, procedures and processes; May be performed by Compliance Analyst | Optional, Capacity Driven | Business |
Data Management Tools
The figure below depicts the Data Management tools as an overlay on top of the Data Management Framework. Similar to how the data management roles were expressed to show where they have the most responsibility or influence, the tools shown below are also placed where the ownership of the tool should exist within the Data Management framework. For example Data Profiling Software which is shown under the Data Quality Management function is an important tool to perform Data Quality Analysis. Although Data Profiling Software is used by roles in other data management functions, it is not central to completing its core function. The organization should consult the Data Quality Management staff when evaluating options for Data Profiling Software as they will have the most experience developing different processes to analyze data.
Figure 4: Data Management Tools
The tools have been specified within the respective data management functions in Figure 4 above to show the link between the roles (people) and the tools they use. For example the tools specified within the Data Security Management function (Network Security Appliances, Secure Transfer Software, IPS/IDS/Firewall Software) are used by the people defined for this function including but not limited to the CISO, Security Staff, Compliance Analyst and Network Administrator.
At the top of the figure are software applications which are used in part by all data management functions. These applications are Report, Project and File Management Software; Performance Monitoring Software to gather Network Traffic, Server, Database and Business Application Statistics; IT Service Management Software to Post and Track Business Requests, and the Business and IT application software used in business and IT management operations.
Ideally, the organization should standardize on the tools the enterprise will use to perform its data management activities. Using Data Profiling Software as the example it is likely data quality analysts, data developers and data architects will all need to leverage the features available within this tool type. When different tools are purchased that meet the same functional needs the financial and human resource burden for the organization grows, which takes resources away from other business or IT opportunities. The organization should look to develop and/or procure tools that enable the required functionality as defined by the cross-functional teams who will utilize it. It is not always apparent who will need to use some data management tools. It will be the responsibility of the data governance team who has been tasked to identify a new IT or Business system tool to bring in a representative from each data management function within the organization to gather all necessary requirements and make the most effective decision.
The list of data management tools shown in the table below and derived from Figure 4 above is provided to begin the conversation regarding the tool set required for an organization with a mature data management program. The Tools provided are not meant to be exhaustive, but to provide examples of the types of tools that can be used within an organization developing their Data Management capability. The high-level costs provided for the tools are estimate ranges as the costs can vary for many of the tools depending on the source; open source versus a well-established product. Your Company should also consider the availability of training and the level to which the tool is accepted in the industry to be assured it can find the staff to use/support them. The cost of the tool is one thing, the maintenance and training support, as well as the cost to hire staff to use it, and the potential cost of recreating the same code when the platform is no longer available, are all important factors when considering price. The ranges below do not account for annual licensing and maintenance fees.
| Tool Type | Description | People/Process | High-Level Cost Estimate (000) per year |
| Network Security Appliances | Network Address Translation and IP Address Management | Network Administrator, Network Security Staff/ Monitor Network Traffic | ~10 – 50 Each Appliance |
| Secure Transfer Software | Enable HTTPS/SFTP/TLS transmission protocols with
Private/Public Key Encryption
|
Network Administrator. Web Developer
Transfer/Transmit Data |
Negligible |
| IPS/IDS/Firewall Software | Network Intrusion Protection/Detection | Network Administrator | ~500 |
| Patch Management Software | Monitor Server, Network, and Application Patch Releases and Install | Network, Server, and System Administrators/ Maintain Infrastructure Hardware, Software Version | Negligible |
| Release Management Software | Migrate applications and code to new release environment
|
Server and System Administrator/ Release, Migrate Code, Application | < 10 |
| Tool Type | Description | People/Process | High-Level Cost Estimate (000) per year |
| Server Administration Software | Provision, Migrate, Take Offline Servers | Server Administrator/Administer Servers | ~100 – 200 |
| Database/File Back-Up and Recovery Software | Back Up Application Environments and Databases | Database and System Administrator/ Back Up and Recover Software | ~10 – 100 |
| Database Management System | Used to develop data (un/structured) storage and retrieval schemas | Database Administrator, Data Developer, Data Warehouse Developer/Develop and Manage Data | ~30 – 1,000 |
| Alert and Notification Software | Generate Automated Alerts and Notifications for IT Systems | Database Administrator, Data Developer, Data Warehouse Developer/ Develop System Automated Error Handling Messaging | < 10 |
| Development Standards and Templates | Documented Templates and Standards for generating application, report and database code
|
Data Developer, Report Writer/ Develop Data and Reports, Dashboards and Visualizations | < 10 |
| Application Development software (Custom) | Application Development Toolkit and Framework for Custom application development
|
Data Developer, Report Writer/ Develop Data and Reports | ~100 – 250 |
| Continuous Integration/Development Software | Development Collaboration Tool for Software Integration and Versioning
|
Data Developer, Report Writer, Data Warehouse Developer/ Develop Data, Report, Warehouse, ODS, OLTP | ~10 – 100 |
| Tool Type | Description | People/Process | High-Level Cost Estimate (000) per year |
| Report/Visualization/Analytics Software | Specialized Report Applications used for Dashboards, Visualizations and Interactive Report (OLAP) | Report Writer, Data Scientist, Data Developer/ Define and Develop Reports, Visualizations and Dashboards | ~10 -250 |
| Extraction-Transform-Load (ETL) Software | Develop data integration and transformation automated processes | ETL Developer, Data Warehouse Developer, Data Developer, Data Quality Analyst/ Extract, Transform and Load Data | ~100 – 1,000 |
| Job Scheduling Software | Create automated job schedules for on-demand and time-sensitive jobs | Data Developer, Data Warehouse Developer, Database Administrator, System Administrator/ Run and Schedule Automated Jobs | Negligible – Built into many Products |
| Document Management Software | Categorize, Manage, Archive, Distribute Corporate Documents | Data Owners, Data Stewards, Technical Writer, Librarian/ Manage Corporate Documents and Records | ~100 – 1,000 |
| Workflow Software | Develop workflow processes with Approval Levels | Data Developers, Business Analysts, Business Unit Managers/ Develop electronic business and IT workflow processes
|
~10 – 250 |
| Digital Signature Software | Enables reduction in management of paper documents requiring signed/authorized approvals
|
Data Developers, Business Analysts, Business Unit Managers/ Develop electronic business and IT signed processes | ~50 -100 |
| Tool Type | Description | People/Process | High-Level Cost Estimate (000) per year |
| Content Management Software | Enable Self-Service change management for web-based content | Data Developers, Business Analysts, Business Unit Managers/ Manage Web Content | ~10 – 100 |
| File Management Software | Enable File Categorization and Access. Manage CRUD privileges for Corporate Files | System Administrators, Business Analysts and Managers / Manage Files and Access Privileges | Built into Windows File Server, SharePoint |
| Archival System Software | Enables the ability to automate processes to archive data and documents based on configured retention policies | Librarian, System Administrators, Business Unit Managers, Database Administrators / Archive Corporate Documents and Data | Part of Document Management Software |
| Data Catalog/Metadata Repository | Used to Develop, Store, Distribute Metadata | All Data Management People / Develop and Communicate Metadata | ~200 – 250 |
| Data Profiling Software | Used to Analyze Data for Quality | Data Developer, Data Quality Analyst, Data Warehouse Developer, Data Analyst / Analyze Data | Negligible; Also Built Into DBMS |
| Test and Test Automation Software | Perform tests on Data and Custom UI Code. Automate Regression Test scripts | Data Developer, Data Quality Analyst, Database Administrator, ETL Developer / Test Code | ~200 – 250 |
| Business and Technical Metadata | Definition of Data from an IT and Business Perspectives to communicate data meaning and requirements | All Data Management People / Design, Develop and Implement Business and IT Architecture, Code and Application User Interfaces. | ~250 |
| Tool Type | Description | People/Process | High-Level Cost Estimate (000) per year |
| Business System Documentation | System and User Manuals | System Administrators, Technical Writers, Business Analysts, Data Owners, Data Stewards / Develop Training and Business and IT Procedures | ~250 |
| Identity/Master Data Management Software | Define and Develop Master (“Golden Record”) and Reference Data | Data Analyst, Technical Writer, Data Developer, Data Architect / Define and Document Master and Reference Data | ~100 – 1,000 |
| Process Modelling Software | Develop Data and Process Flow Diagrams; Data Lineage Models | Data Architect, Database Administrator, Data Developer, Data Warehouse Developer, Quality Analyst, Business Analyst, Technical Writer/ Design Data and Business Process Flows | < 10 |
| Data Modeling Software | Develop Entity Relationship Diagrams for Data Analysis and Application, Architecture Design | Data Architect, Database Administrator, Data Developer, Data Warehouse Developer/ Design Data Models and Architecture | < 10 |
| Business/IT Data Standards | Documentation used to develop data, application interfaces per business and IT standards. Used to perform Data Quality Analysis | Data Developer, Data Quality Analyst, Data Architect, Data Warehouse Developer, ETL Developer/ Profile Data, Data Quality Analysis, Develop Data Architecture Models | < 100 |
It is not necessary for an organization to purchase all of the tools that are described here to enable the respective data management function. Tools, similar to the People and Processes described in this document, must be associated with a business purpose prior to consideration. This does not mean that IT loses its authority to purchase IT specific tools, rather that all tools that consume the organization’s financial resources should be associated with a business purpose. IT tools that monitor system performance or protect data assets at the network level fit in with business goals of ensuring the performance and security of its data assets. One best practice your Company should enforce to become good stewards of the financial resources it invests in data management tools is to evaluate the purchase by the data governance team and require centralized authorization to ensure the purchase aligns with business strategy and is not redundant.
Data Management Processes
The Data Management Processes shown in Figure 5 below are depicted within the Data Management Framework and are aligned with their respective data management functions. By overlaying the data management framework diagrams for people, tools and processes it will become easy to see how each of the elements relate to support data management capabilities.
The processes documented below are not meant to be exhaustive. Based on the organization’s structure and scope of work, the number of processes must be tailored to the business goals and requirements. The processes provided are high-level processes that are considered essential within the function for which they are listed. If the organization does not have a business need to implement a specific data management function then those processes are not necessary. Similar to planning for staff when developing the organization’s data management program, the number of processes also depends on the specialization required. Where there is a need to break down capabilities to a fine grain level and staff become more specialized, the number of processes will increase. Any of the processes listed that the organization deems unnecessary should be documented as exceptions by the Data Governance committee making that determination. The agreement by business and IT to defer a process provided in the figure below can affect the effectiveness of the organization to build data strategies that align with business strategies. If the business later determines the capability is necessary based on a new strategy or objective, business and IT will be informed by the previous decision that the effort to implement the new data requirements can’t begin until the data management function exists. The goal is to be capable of supporting the business strategies and avoid situations where delays in delivering on the strategies derive from poor data management planning.
The Data Management processes become clearer when analyzed within the context of an agreed-upon, well-defined purpose statement. The execution of the processes will be more efficient and effective when based on well-defined business and IT policies, procedures, processes and standards, and applied only when necessary to achieve a business need.
Figure 5: Data Management Processes
The processes shown in Figure 5 above represent the essential processes within their respective functions. A process is not complete unless there are people assigned to the process, who are trained and experienced in performing the process steps, have knowledge and experience using the tools necessary to perform the process, and there is adequate documentation in the form of policies, procedures, and standards to inform the process. The goal for the organization is to develop processes that are repeatable and performed in a consistent fashion. Depending on the structure of the organization and the type of products and services it delivers each of these functions can have tens to hundreds of processes defined. The first step in defining the function’s processes will be to determine the business processes that must be enabled by data and systems, and to associate the business process requirements with one or many of the data management functions. Furthermore, depending on the reusability of the Data Management function processes, they should be packaged and provided to Business as a Data Management Service. Examples of packaged services include data requirements analysis, data definition and modeling, and application release services.