Optimising Dynamics 365 F&O Storage Costs: When to Purge, Archive, or Expand
Yesterday morning started with a call from my CEO friend. The email from Microsoft landed in his inbox: "Your Dynamics 365 environment has exceeded 80% of allocated storage capacity." For his retail business, this wasn't just a warning - it represented a potential $35,000 annual increase in cloud costs if left unaddressed.
Data growth in Dynamics 365 F&O environments is inevitable. Every transaction, every document attachment, every workflow history item - they all consume space. Yet I've found that most organisations lack a coherent strategy for managing this growth, often defaulting to simply purchasing more storage when limits are reached.
After helping dozens of Australian companies optimise their D365 F&O storage footprint, I've developed a systematic approach to determine when you should purge, when you should archive, and when expanding storage is genuinely the right economic decision. Let me share what I've learned.
Data growth in Dynamics 365 F&O environments is inevitable. Every transaction, every document attachment, every workflow history item - they all consume space. Yet I've found that most organisations lack a coherent strategy for managing this growth, often defaulting to simply purchasing more storage when limits are reached.
After helping dozens of Australian companies optimise their D365 F&O storage footprint, I've developed a systematic approach to determine when you should purge, when you should archive, and when expanding storage is genuinely the right economic decision. Let me share what I've learned.
Understanding D365 F&O Storage: What You're Actually Paying For
Before diving into strategies, let's clarify what storage actually means in the context of Dynamics 365 F&O, as there are several distinct components:
Database Storage: Contains your transactional data, master records, and system tables. This is typically the largest component and grows as your transaction volume increases.
File Storage: Houses documents, attachments, and media files linked to records in your system.
Log Storage: Maintains system logs, audit trails, and workflow history.
The standard allocation included with D365 F&O licences typically provides:
Database Storage: Contains your transactional data, master records, and system tables. This is typically the largest component and grows as your transaction volume increases.
File Storage: Houses documents, attachments, and media files linked to records in your system.
Log Storage: Maintains system logs, audit trails, and workflow history.
The standard allocation included with D365 F&O licences typically provides:
- 10GB database storage per tenant
- 20GB file storage per tenant
- Additional incremental storage based on user count
- Database storage: $48 AUD per GB per month
- File storage: $2.50 AUD per GB per month
The True Cost of Data Retention: Beyond Storage Fees
Storage fees are just the beginning. Retaining unnecessary data in your production environment imposes several hidden costs:
Performance Degradation:
Every gigabyte of unnecessary data can impact system performance, particularly for operations like month-end closing, MRP runs, and complex reporting.
Backup and Recovery Time:
Larger databases mean longer backup times and extended recovery periods during outages - potentially adding hours to your RTO (Recovery Time Objective).
User Productivity Impact:
Searching through years of unnecessary historical data increases the time required to find relevant information.
A retail client discovered that purging three years of obsolete transaction data not only saved $28,000 in annual storage costs but also reduced their month-end closing process by 40% - a far more significant business benefit than the direct cost savings.
The Decision Framework: Purge, Archive, or Expand
Based on dozens of data optimisation projects, I've developed a decision framework that helps determine the appropriate strategy for different data categories:
WHEN TO PURGE: Deletion Candidates
Data that meets the following criteria is typically best handled through permanent deletion:
Temporary Data:
- Temporary import tables
- Integration staging tables
- ETL transformation tables
- Batch processing temporary data
Redundant Data:
- Superseded document versions
- Duplicate reports and exports
- Debug logs and traces older than 30 days
- Workflow history for completed workflows older than 90 days
Error Data:
- Failed integration records after resolution
- Error logs older than 90 days
- Failed batch jobs after review
A logistics company I advised implemented automated purging of integration staging tables, reducing their database size by 22GB and saving approximately $12,700 annually with minimal business impact.
WHEN TO ARCHIVE: Cold Storage Candidates
Data with these characteristics should typically be moved to lower-cost archival storage:
Historical Transactional Data:
- Closed financial periods beyond statutory retention requirements
- Completed sales orders beyond warranty periods
- Purchase order history beyond vendor agreement terms
- Completed production orders beyond quality hold periods
Reference-Only Documents:
- Historical price agreements
- Expired contracts and agreements
- Previous versions of master data
Audit and Compliance Data:
- Detailed audit logs beyond active monitoring periods
- System access logs beyond security review periods
- Compliance documentation beyond immediate reference needs
A food manufacturing client implemented an archiving strategy for completed production orders, moving 56GB of data to cold storage. This reduced their database footprint by 35% and saved approximately $32,300 in annual storage costs while maintaining data accessibility for compliance purposes.
WHEN TO EXPAND: Retention Candidates
In some cases, paying for additional storage is the right business decision. Typical scenarios include:
Active Business Data:
- Current and recent fiscal year transactions
- Open orders and active processes
- Master data and configuration
Business Intelligence Assets:
- Data required for trend analysis
- Information needed for active forecasting
- Customer buying pattern history used for current sales initiatives
Legal and Compliance Requirements:
- Records required by ATO or ASIC
- Industry-specific retention requirements (e.g., APRA for financial services)
- Data subject to active legal holds
High-Retrieval Information:
- Frequently accessed historical information
- Data that requires sub-second retrieval performance
A healthcare supplier determined that maintaining three years of sales history in the production database was justified by the frequency of reference and analytical value. For this 78GB of data, they accepted the $44,900 annual storage cost as a necessary business expense.
Practical Implementation: Tools and Techniques
With the decision framework established, let's explore specific implementation approaches:
Data Purging Strategies
Batch Job Scheduling: D365 F&O includes several standard data cleanup batch jobs that can be scheduled to run automatically:
Data Management Framework: The D365 Data Management Framework can be leveraged for more complex deletion scenarios, particularly when referential integrity must be maintained across related tables.
A manufacturing client implemented weekly scheduled purging of completed workflow history, reducing their database growth rate by 3.5GB per month and saving approximately $2,000 monthly.
- Document Handling cleanup
- Workflow history purge
- Batch job history cleanup
- Electronic reporting logs cleanup
Data Management Framework: The D365 Data Management Framework can be leveraged for more complex deletion scenarios, particularly when referential integrity must be maintained across related tables.
A manufacturing client implemented weekly scheduled purging of completed workflow history, reducing their database growth rate by 3.5GB per month and saving approximately $2,000 monthly.
Data Archiving Techniques
Built-in Archiving Functionality: D365 F&O includes standard archiving capabilities for certain modules:
- General ledger transaction archiving
- Inventory closing archive
- Production order archiving
- Sales and purchase history archiving
- SQL Server Stretch Database (moving cold data to Azure SQL)
- Azure Data Factory pipelines for structured archiving
- Azure Storage for document and attachment archiving
- Icertis (formerly Annata) Data Archive for D365
- To-Increase Data Archive
- Binary Stream Data Management Suite
Storage Expansion Considerations
Incremental Purchasing: When additional storage is necessary, consider:
Licence Right-sizing: Sometimes, adjusting your licence model can be more cost-effective than purchasing additional storage separately.
A retail client negotiated a 20% discount on storage expansion by committing to a 12-month projection of their storage needs, saving approximately $9,300 over the contract period.
- Purchasing in smaller increments to match actual growth
- Negotiating volume discounts for predictable growth
- Exploring CSP (Cloud Solution Provider) options which may offer more flexible terms
Licence Right-sizing: Sometimes, adjusting your licence model can be more cost-effective than purchasing additional storage separately.
A retail client negotiated a 20% discount on storage expansion by committing to a 12-month projection of their storage needs, saving approximately $9,300 over the contract period.
Creating a Data Lifecycle Management Strategy
Rather than addressing storage issues reactively, establish a comprehensive data lifecycle management strategy:
Data Classification Exercise
Conduct a thorough classification of your D365 F&O data based on:
- Business value
- Access frequency
- Performance requirements
- Compliance obligations
- Cost of storage vs. cost of retrieval
Retention Policy Development
Develop formal retention policies for each data category:
- Transaction data (by module)
- Master data
- Documents and attachments
- System logs and audit trails
- Workflow history
Automation Implementation
Implement automated processes for executing the policies:
- Scheduled batch jobs for regular purging
- Periodic archiving routines
- Exception handling for policy overrides
- Documentation and audit trail of data lifecycle actions
Monitoring and Governance
Establish governance mechanisms:
- Regular storage utilisation reviews
- Growth trend analysis
- Policy compliance monitoring
- Cost impact assessments
A professional services firm implemented a comprehensive data lifecycle strategy that reduced their annual storage costs by $52,000 while improving system performance and maintaining compliance with their document retention obligations.
Real-World Cost-Benefit Analysis: By Industry
Different industries have unique data patterns and retention requirements. Here are examples of optimised approaches I've helped implement:
Manufacturing
Pain point: Rapid database growth from production orders and quality data
Optimised approach:
Optimised approach:
- Purge: Test results for non-critical components after 90 days
- Archive: Completed production orders older than 18 months to Azure Storage
- Retain: Quality data for critical components for the full product lifecycle
Retail
Pain point: Massive transaction volume with seasonal spikes
Optimised approach:
Optimised approach:
- Purge: Point-of-sale detailed logs after 60 days
- Archive: Transaction details older than 12 months, maintaining summarised sales data
- Retain: Customer purchase history for active customers for loyalty program
Financial Services
Pain point: Stringent compliance requirements with long retention periods
Optimised approach:
Optimised approach:
- Purge: System processing logs after compliance review
- Archive: Customer transactions to immutable storage after 18 months
- Retain: Account master data and compliance-related transactions for statutory periods
Final Thoughts: A Proactive Approach
Data storage costs are often treated as an inevitable tax on digital business operations. My experience suggests otherwise - with a strategic approach, most organisations can reduce storage costs by 30-60% while improving system performance and maintaining compliance.
The key is shifting from reactive expansion to proactive management. By treating data as an asset with a defined lifecycle, you can maximise its business value while minimising its infrastructure cost.
Remember that storage savings compound over time. A 50GB reduction doesn't just save you $28,800 in year one - it saves that amount every year, plus eliminates the growth that would have occurred on that data volume. Over a five-year period, seemingly modest storage optimisation can easily represent $250,000+ in direct savings, not counting the performance and productivity benefits.
The key is shifting from reactive expansion to proactive management. By treating data as an asset with a defined lifecycle, you can maximise its business value while minimising its infrastructure cost.
Remember that storage savings compound over time. A 50GB reduction doesn't just save you $28,800 in year one - it saves that amount every year, plus eliminates the growth that would have occurred on that data volume. Over a five-year period, seemingly modest storage optimisation can easily represent $250,000+ in direct savings, not counting the performance and productivity benefits.