It has been clear that cancer-associated fibroblasts (CAFs) in the tumor microenvironment play an important role in pancreatic ductal adenocarcinoma (PDAC) progression. However, how CAFs relate to the patients' prognosis and the effects of chemoradiation therapy (CRT) has not been fully investigated. Tissue microarrays (TMAs) representing 167 resected PDACs without preoperative treatment were used for immunohistochemical studies (IHC) of palladin, α-smooth muscle actin (SMA), and podoplanin. Correlations between the expression levels of these markers and clinicopathological findings were analyzed statistically. Whole sections of surgical specimens from PDACs with and without preoperative CRT, designated as the chemotherapy-first group (CF, n = 19) and the surgery-first group (SF, n = 21), respectively, were also analyzed by IHC. In TMAs, the disease-specific survival rate (DSS) at 5 years for all 167 cases was 23.1%. Seventy cases (41.9%) were positive for palladin and had significantly lower DSS (p = 0.0430). α-SMA and podoplanin were positive in 167 cases (100%) and 131 cases (78.4%), respectively, and they were not significantly associated with DSS. On multivariable analysis, palladin expression was an independent poor prognostic factor (p = 0.0243, risk ratio 1.60). In the whole section study, palladin positivity was significantly lower (p = 0.0037) in the CF group (5/19) with a significantly better DSS (p = 0.0144) than in the SF group (16/22), suggesting that stromal palladin expression is a surrogate indicator of the treatment effect after chemoradiation therapy.