Database Normalization And Salesforce - Ponderings

One of the biggest challenges in moving from a highly structured Oracle environment into Salesforce is how denormalized Salesforce's querying / reporting / data-viewing limitations encourage one's data model to be.

This particularly seems to be an issue in higher education. Like banking or healthcare, most of its data is rigid, with the typical business user's number one concern being easily expressed as, "Please don't mess up my data." Overall, higher education's business needs beg to have a university's databases insanely normalized.

Except on the fringes, that is. On the fringes, a university's interactions with the outside world stop being about individual human beings who care about "their" data "not being messed up." Prime "fringe" examples include:

1. Email-address-based marketing, which is free and great, but hard to pin down to an actual person because people share
2. Continuing education and seminars, where it's not really a big deal to swap attendees - if it's corporate training, who at the sending company cares who comes as long as someone gets trained?

These "fringe of the university's business" use cases, plus staff's desire to see the two sets of data married, seem to tempt universities to look at databases beyond their 20-year-old ERPs. (Of course such "data marriages" also desire the tall order of doing so without polluting vital information like the admission-graduation pipeline, employment history and paychecks, donation history!)

• Unfortunately, this marriage means you're back to squeezing "fuzzy" data into a normalized structure. Which, in theory, you could do in your ERP if it gave you an easy way to add tables (and user interfaces to those tables).
• Then there's the route of struggling to squeeze highly-normalized "vital" data into a "flat" structure like Salesforce encourages. Unfortunately, this can mean you're always fighting inaccuracy, lag, duplication...

Integrating Salesforce and our ERP (plus a few new data stores) is a much longer road than I was expecting, and it all keeps coming down to issues with matching a data model to the real world.

Banner is an amazing set of highly normalized tables for "don't lose my data" core university functions.
(If you get a look at the EnrollmentRx plugin for Salesforce as a Banner user, you'll be amazed - it's basically SPAIDEN/SRARECR/SAAADMS/SOATEST built into a newer database and restricted to "1 Contact/SPAIDEN record per email address." Everyone re-invents the wheel for higher ed because, like banking, some things just don't change.)

I think Sungard/Ellucian really missed the boat here.

They could have raked in the big bucks if they'd "taken care of the fringes" and:

1. Given Banner just a few tweaks for easily adding/removing tables (and corresponding user interfaces)
2. dded a good way to upsert the database 1-record-at-a-time
   (e.g. web forms - as FormAssembly's Salesforce connector proves, you really just need to have a user interface that lets a form-handler-configuration-managing user build complex SQL queries sorted by LastModifiedDate descending - not everything has to be "common-matched" - similar idea for fast deduplication ... DemandTools, which is to PL/SQL what Cognos is to SQL, exists because a good CRUD API to the Salesforce database exists)
3. Engineered a great "over"-layer for email tracking and sending - especially by admissions departments.

For example, what if there were some sort of API that let you quickly create/update/delete SPACMNT records from what's otherwise basically a Cognos report? Like, a little pencil-shaped "edit" button near the list of comments, but otherwise, nothing is editable - it's just the read-only report (doing nice things like hiding details of inactive records and only showing you the most-recent SRARECR/SAAADMS/SGASTDN data available, as determined by the report-author)? Or a little "edit" button next to "status" that lets you pick from a picklist, and then when you commit it, the "interactive report" author has programmed logic to update SRARECR/SAAADMS/SGASTDN accordingly? That's basically what one of our main departments wants. A "collapsed, current, relevant" view of dispersed back-end data, with a few "edit" buttons interspersed so they don't have to surf through the normalized tables themselves to make updates.

The other major reason we tried Salesforce was because it had plugins for the web/email era and Banner didn't. (And a final reason was to accommodate those "fuzzy"-work departments that were using spreadsheets, Access databases, Rolodexes, etc.)

I often wonder where we'd be if we'd been able to do all this development straight into our highly-normalized ERP instead of half-rebuilding Banner inside of Salesforce.

What would we have done if we could have web'd/emailed/custom-tabled/easy-writeback'ed our existing database + data model instead of replicating our tried-and-true data model inside of Salesforce? What could we have done if Ellucian had tacked on the user-friendly concepts that Salesforce made popular instead of leaving their customers in the 1990s?

Salesforce's good luck and Ellucian's loss, I suppose. But I wish I often wish I could have my cake and eat it too - full normalization support and modern-database flexibility.

Since I can't, I still believe that the ideal for universities is to roll out Salesforce "from Rolodex to ERP," not "from ERP to Rolodex."

6 Ways To Code Field Value Changes / DML In SalesForce

Back when I first started designing a solution to the problem of syncing ERP "Admission" records to SalesForce "Opportunity" records, I approached it with the idea that "synchronization" means "examining both objects" for sameness or differences.

That's what I would have recommended in Oracle. We would have written a PL/SQL stored procedure or anonymous procedure with an IF (criteria) THEN (action) structure. We would have scheduled it to run as often as we expected our data to change.

 

However, I couldn't find the equivalent of a PL/SQL stored procedure - something you can invoke just because you feel like it - in SalesForce.

Sure, there were Methods in Apex Classes, but I couldn't figure out how to invoke them at will.

I wanted this functionality because a record from either object - "Admissions" or "Opportunity" - could change in a way that creates or breaks a "match" between the two objects.
It takes two to tango when you're talking about "synchronization."
Triggers (both in Oracle and in SalesForce) can only react to one type of object changing its value. A single trigger can't be progrmamed to fire based on updates to records of either object type.

 

After I posted my dilemma on the SalesForce Success Community, James Loghry of EDL Consulting provided this wonderful summary of the ways you can programmatically / batch change the value of one field based on the value of another:
(text in {}'s mine)

 

1. Apex Triggers / {Workflow Rules} / Process Builder for handling updates / logic when a record {of a given object type} is created or updated.
    
2. Batch Apex which can be executed manually or on a scheduled, periodic basis.
    
3. Web Services (REST or SOAP) to update the records from either the same Salesforce instance or an external data source.
    
4. A middleware solution (for instance Oracle Fusion / BPEL or Jitterbit) that interacts with Salesforce.
    
5. You could also manually use tools like Dataloader / {DemandTools} to update the records manually.
    
6. Depending on your relationships and requirements, you could potentially use formula / {rollup summary} fields between the detail records and master contact record instead of performing "DML" transactions (record creates, updates, deletes, undeletes, merges, etc.)
   {How cool would it be if there were such a thing as "SOQL fields"? *alas*}

 

Batch Apex sounds like a very promising way to "keep thinking about synchronization the way I always have in Oracle," although I'm disappointed by the language implying that you can only have 5 jobs scheduled at a time. I wonder how much functionality you can safely cram into a single job...

 

In the meantime, because our consulting partners are skilled at writing Apex Triggers, we have tried to ask ourselves where a trigger could handle most of the work. In the case of synchronizing "Admission" and "Opportunity" records, we expect that 90% of the time, changes to the "match" between records will come from changes to Admission (ERP) records.

Our department's SalesForce users will simply have to be warned that they play with certain fields on "Opportunity" records at risk of their own confusion (about why the data didn't re-synchronize itself).

 

I'm daydreaming of more, but it's a good start in a fast push to go live.

Rolling Up A Single Record To Its Master Object

Rollup fields are the other of the two types of "calculated value" field allowed in SalesForce.

When you have a "Master-Detail" relationship between two objects in SalesForce, you can add a Rollup Field to the "Master" object that shows the result of an aggregation (count/min/max/etc.) on the values found in a single "column" of the "Detail" object.

But you don't have to use rollup fields exactly as they were intended!

 

Today, I was asked to add a field to "Admissions" called "Application Form Received Date."

After poking around in reports from our legacy system (Banner), I discovered that we calculate this date by looking at the "Received Date" of the checklist item called "Application Form" that is attached to the a given "Admissions" record.

In both Banner and the SalesForce "dumping ground custom objects" for imported Banner data, each application document that we have received is stored in its own record of a "Checklist" object.  "Checklist" is at the "detail" end of a Master-Detail relationship with "Admissions."

It just so happens that in our data model, no "document type" may be used twice in the same set of "Checklist" records.  (Or, in our classic "tines of a garden rake" example, no color of tine may be used more than once on a given rake.)

 

So back to the analogy of children drawing on a garden rake (copying details) as they build it:  the rule here is that if you see an orange tine, you have to copy whatever is doodled on it onto the handle of the rake.

A formula field won't help you implement this rule. Formula fields don't know what to do when they have multiple rows to choose a value from, so they can only copy field values "from handle to tines" (from Master to Detail or from LookedUp to LookingUp).

 

However:

• If you want to copy NUMERIC or DATE data "from tine to handle" (from Detail to Master),
• And you know that you can specify "which color of tine" by inspecting the values of fields on the "tine" (except formula field values),
• And you know that only "1 tine of that color" will exist...

...Then doing a MAX or MIN "Rollup Field" on the "tine" (Detail) field will make an exact copy of it on the "handle" (Master).

 

After all, the solution to "What is the biggest number between 4 and 4?" is ... 4.

 

To build the "Application Form Received Date" field on the "Admissions" object, I added a Rollup Field defined as the maximum "Received Date" among all eligible Checklist items. I then filtered the definition of "eligible Checklist item" to records where "Document Type" = "Application Form" (knowing that there would be only 0 or 1 such records).

 

Yay for kludges! (I think I want to print that and hang it on my wall.)

A Formula Field To Pick The Best Custom Object Among Many

Formula fields are one of two types of "calculated value" field allowed in SalesForce. (The other type being lookup fields, which do not allow any calculations besides aggregations.)

As I mentioned, while most universities choose either custom "Admission" objects or the SalesForce native "Opportunity" object to represent the many-to-one nature of people trying to matriculate to their school, we are forced to live in, and synchronize, both models.

Keeping this data synchronized requires code to perform a cardinality reduction. I used a formula field to do the heavy lifting.

 

Note: If you want to skip the background and go straight to the formula field's source code, click here.

 

Imagine that you're building a garden rake with tines at both ends of the handle. (The handle is a particular "Contact." Each tine at 1 end of the rake represents "Admissions" records. Each tine at the other end of the rake represents "Opportunity" records.)

• You allow the neighbor's children to add tines to the "Admissions" end of the rake.
  (This is like the Central Admissions Office adding records about paperwork coming into their office from the Contact.)
  • The neighbor's children are allowed to draw pretty pictures on the "Admissions" tines before they attach them to the rake.
    (This is like the Central Admissions office recording the details of that paperwork.)
• You allow your own children add tines to the "Opportunities" end of the rake.
  (This is like your department adding records about their intuition that someone is interested in matriculating based on phone calls, event attendance, etc.)
  • Your are allowed to draw pretty pictures on the "Opportunities" tines before they attach them to the rake.
    (This is like your office recording the details of when they think the person would want to matriculate and other relevant information.)

You have a special rule for your children (working on the "Opportunities" tines): Although they're not allowed to add tines to the "Admissions" end of the rake, periodically they must look at the "Admissions" tines that have already been attached. If they see that there is an "Admissions" tine attached to the rake that is the same color as one of the "Opportunity" tines that they have attached, they must copy over all of the pretty pictures from that "Admissions" tine onto their matching "Opportunity" tine.
(This matches our business rule that SalesForce must push all data from "Admissions" objects to their matching "Opportunity" objects so that our recruiters can see everything they need to see on "Opportunity" objects.)

 

That's the kind of environment we're developing in.

There's just one more catch:
Our Central Admissions Office is allowed to add multiple tines of a given color, whereas our Departmental Recruiters must make every tine at their end a unique color. (The color of a tine is analogous to "Program Of Interest.")

The ERP requires that every Person-Program-Term combination be given its own record. Our recruiters don't care about that level of detail - they just want to see the "most relevant" information for a given Person-Program combination.

 

So we have a cardinality reduction problem on our hands. This lets the "doodle-copying" code know which "blue tine" on the "Admissions" side of the rake is the "best" so that we can copy its doodles over to the "blue tine" on the "Opportunity" side of the rake. Otherwise, the "doodle-copying" code will get confused.

 

Here's how I solved it: (update here)

First, I created a formula field on Admissions called Record_Type_Prioritizer__c.

I should explain that: I've oversimplified our problem up above. We actually have 2 similar record types coming in from the ERP - "Recruit" and "Admissions." So Record_Type_Prioritizer__c has objects label themselves 9 (if Admissions), 8 (if Recruit w/ an "app started" flag), or 7 (plain-old recruit).

Next, I created a formula field on Admissions called Winner_Picker__c. (It's also on "Recruit.")

Here is how I described Winner_Picker__c to end users so they could review my logic:

WinnerPicker is a calculated numeric field. First, it disqualifies any "past" records or "withdrawn" by being blank for such records. Then it ranks Admissions records as best, Recruit records with an Application Started? flag as 2nd-best, and other Recruit records in 3rd place. Within each of those categories, it ranks records in descending order from soonest term (best) to futuremost term (worst). If there are still any ties, it breaks them "coin toss" style by ranking them in descending order according to their unique record IDs from Banner.

Here is the math behind this formula:

• Return null for any term that has already started or any Admissions record with an inactive status
• Admissions = 9,000,000; Recruit+Flag = 8,000,000, Recruit-flag = 7,000,000.
• Subtract term code (a 6-digit #; subtraction because we care about sooner terms more than later ones as long as they’re in the future).
• Multiply that 7-digit # by 1,000,000 to add a comfortable amount of zero-padding at the right.
• Add 900 or 800 or 700 (Adm / Recr+Flag / Recr) because the last 2 digits are unique within, but not across, sets of Admissions/Recruit records for a person.
• Add the 2 unique digits not already covered in the previous code from the Admissions/Recruit record's primary key in the ERP as a tie-breaker.

Finally, here is the source code of Admissions.Winner_Picker__c:
Note that Admissions's "Term" field is a Lookup Field to a "Term" object. "Term" records are full of fun formula fields themselves! They provide data about a given term in many formats: as a sortable number, as a comparison of its start date to today, etc.

IF(
Term_Lookup__r.Future__c ,


IF(
OR(
ISPICKVAL(Status__c, "Deferred"),
ISPICKVAL(Status__c, "Rejected"),
ISPICKVAL(Status__c, "Withdrawn")
)
,
null,
(((Record_Type_Prioritizer__c * 1000000) - Term_Lookup__r.Term_Code_Numeric__c)* 1000000)
+
((Record_Type_Prioritizer__c * 100) + VALUE(RIGHT( Banner_Application_ID__c , ( (LEN(Banner_Application_ID__c) - 6) - LEN(Formula_Contact_PIDM__c) ) )))
)


, null
)

 

 

The Result:

Each "Admission" record looks at its field values and suggests its "self-worth" as a number.

This "self-worth" number is guaranteed to be a unique value among all Admissions+Recruit records with the same Person+Program combination.

Thanks to this formula field, you simply have to perform an "IS NOT NULL" and a MAX() to retrieve the best ERP-imported record for a Program.
(Or, in other words, "The best blue tine on the rake.")

It saved us a lot of time during a Requirements Analysis meeting with our Partners doing trigger development work to say, "Don't worry about the cardinality reduction logic - we got it - just MAX() this field for the Person+Program."

 

Now if only you could use a Rollup Field to MAX() Formula Fields ... Don't forget to click here and upvote! :-)

Background: Our Recruiters Rely On Opportunity Objects

A bit of background on the code we're developing:

The records storing the details of a person's interest in matriculating to a higher education program form a many-to-one relationship with that person.

What does "many-to-one" mean? Here's an example:

I can be interested in matriculating to my undergraduate studies at Harvard, {get rejected or matriculate and hopefully graduate}, 5 years after that decide I want a master's degree in English there, {get rejected or matriculate and hopefully graduate}, and 10 years after THAT decide that what I would really like is a master's degree in French there.

"Records about my interest in Harvard" are "many-to-one" because there is only 1 of me, but there are 3 separate "interactions" I've had with Harvard about my interest in it.

To track these interactions, my peers who went to DreamForce tell me that Admisisons offices using SalesForce typically break down into two categories:

1. Those that use SalesForce's native "Opportunity" object to track "interest in matriculating."
   • FYI, using "Opportunity" as if it were in a many-to-one relationship with "Contact" involves populating a field on Opportunity called "Primary Contact Role" with the "Contact" record of the person interested in matriculating.  Apex Triggers can then grab that record and treat the relationship like a "lookup" relationship.
2. Those that create custom "master-detail" objects hanging off of "Contact" objects to track "interest in matriculating." Such objects are often called something like "Admission."

We're an "Opportunity" shop (model #1).

But Banner, our ERP, works on model #2 ("Admission" records).

And we're currently in a position where the rest of the school is maintaining those records in Banner on our behalf, rather than us doing everything in SalesForce the way most "Opportunity" shops would.

So we have "Admission" detail objects hanging off of "Contact" in SalesForce AND "Opportunity" objects acting as if they hung off of "Contact."

The "Admission" objects get populated every morning by a PLSQL dump to .CSV files and automated data loading of those .CSV files into SalesForce.  (We know, we know.  We're looking into JitterBit.)

Finally, we don't have the resources we need, in the timeline we need them before "go-live," to restructure the "Admission" object or its corresponding PLSQL code into something that our recruiters can easily work off of. (We would want more "collapsing" of Banner data to be done before import so that back-to-back applications to the same program update old records rather than creating new ones.)



We're an "Opportunity shop" because:

• "Opportunity" provides us a "blank slate" in this environment where "Admission" is set in stone.
• "Opportunity" also plays better with SalesForce plugins like the mass-mailing software Pardot.

We just need to figure out how to "collapse and copy" relevant data from "Admission" records into the "Opportunity" records that are equivalent to them.

Much of the code I'll cover in upcoming posts has revolved around our need to act like an "Opportunity shop" without ignoring the contents of data feeds that think we're an "Admissions object shop."

Stay tuned.

Welcome To "Oracle 2 SalesForce!"

Welcome!  My name is Katie, and I began my journey as a SalesForce system administrator in November 2014.

I work in a large academic department of a mid-size university.  We were the first department to purchase a SalesForce org, although the school at large has now purchased one as well.

Before taking my current position, I worked at the same university managing data entry into, and Cognos Report Studio reporting out of, our Entrprise Resource Planning (ERP) system, Banner, on top of an Oracle back-end.

Although I have not written PL/SQL to control DML (INSERT, UPDATE, DELETE) professionally, I am familiar with it through graduate school and personal experimentation.  In other words, I'm just dangerous enough to think I can talk about my blunders through enterprise-grade database systems without having the chops to really impress people who come here for the word "Oracle."  Hope you enjoy your visit anyway!

The university's ERP is still a major source of information in our SalesForce org, but not the only one.  As of the founding of this blog in March 2015, we have some interesting constraints on our work (such as working around a fixed-structure daily import until we have JitterBit to replace the PL/SQL behind it, and such as the rest of the university still doing primary data entry into the ERP).

I have found it quite a paradigm shift to design solutions to problems integrating our ERP data into our SalesForce org for the Admissions department--one I'm still "shifting" through every day.

SalesForce Success Communities, YouTube videos, and StackExchange boards have been invaluable, but sometimes I feel a bit lonely as an "Oracle person in a SalesForce world."

So...if you do too, welcome!  As I come up with "SalesForce-y" solutions to problems I used to know how to solve in Oracle, I plan to share them here.  May you find them useful in your quest to master your new world as well.  :-)

-Katie