I'm wondering what Tascman and Capt Dan have to say.
I say you're making your life too difficult.
My suggestion - at least to get things set up and working properly: use the INPUT BYPASS method to get signal to Protools. Then, ensure that Ptools' channel outputs are correctly assigned back to your chosen fader modules on the DM.
Example:
Let's say you're tracking a guitar; it's plugged into the DM's Input (M/L)#1. Hit ROUTING/OUTPUT SLOT (tab #2 at the bottom of screen). On the right of screen select "M/L". Then, on left side of screen, Pick Input #1 and dial thru choices 'till you see: 'M/L1". Hit Enter. What you've just done is tell the DM to send the gtr's signal directly from the preamp to the Firewire's Channel #1. This means that the gtr's signal volume will ONLY be affected by the DM trim pot and/or the outboard device plugged into M/L#1.
Next, you can monitor that gtr on any fader module - including those on the 33-40 layer. So, let's pick CHannel 40 as the monitor channel. Hit: ROUTING/Input; SELect Fader Module 40, and after selecting M/L on the right side of screen, dial thru the choices until M/L#1 appears. Hit Enter to confirm. What you've just done is assign the signal from the preamp controlling the gtr to Fader Module. #40. (I like to keep my Channels 33-38 open for efx returns.) Don't forget: Assign #40 to the STereo Bus (hit STereo button atop left side of console!)
Finally - in Protools: Assuming you've set up your ASIO/Firewire correctly, you should be able to click on PT's Input Icon on any selected track. You'll see the entire 32 input choices in the menu. So - you've sent signal down Firewire Channel #1, tell PT to RECEIVE that signal on - say - PT's Track #7. Choose PT Track #7, hit the input icon, then select Input #1. Then, you need to tell PT to send that track's signal BACK to the DM. Hit PT Channel 7's OUTPUT icon, and choose a Firewire pathway - say - Mono Output #5.
Now - on the DM: Decide which fader module you'd like to hear AND control PT's gtr track. (Everybody sets up their boards differently. I like to keep drums, percussion, keys, gtrz, efx, etc in their own adjacent groups. Makes mixing easier. However - faders can be swapped, moved, deleted, or exchanged at any time.)
Let's say you want to use DM Fader Module 18. So: Hit ROUTING/Input (tab 1); select Fader 18, and dial through the choices until INPUT #5 appears. Hit ENTER.
What you've just done is assign PT's Firewire output to DM's Fader #18. Finally:
be SURE that Fader #18 is assigned to the STereo Bus! Hit the STereo button atop the left side of console in the Bus Array.
CAUTION!!! When actually tracking any part, be sure to MUTE ProTools' output module (in this case Fader 18 or PT's Remote Fader). Otherwise, you'll get annoying latency. This is why you've delegated DM Fader 40 for tracking/monitoring: NO LATENCY!. Then, for playback, do the reverse: mute 40, and unmute 18.
That's all there is to it. It WILL work if you follow these instructions. And - when you've got all working, SAVE the setup to a Snapshot or to a File for successive projects.
CaptDan